[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:00:30 dist-test-slave-5n86 krb5kdc[31715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:00:30 dist-test-slave-5n86 krb5kdc[31715](info): set up 2 sockets
Sep 25 05:00:30 dist-test-slave-5n86 krb5kdc[31715](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 05:00:31.629928 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.006s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:00:31 dist-test-slave-5n86 krb5kdc[31715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:00:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:00:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:00:31.824090 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:37473
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:37473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:32.135679 31728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:32.135943 31728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:32.136029 31728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:32.142884 31728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:00:32.142983 31728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:32.143028 31728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:32.143085 31728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:00:32.143134 31728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:00:32.152374 31728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:37473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:37473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:32.153640 31728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:32.154870 31728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:32.166507 31736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:32.166554 31734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:32.166613 31733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:32.167171 31728 server_base.cc:1034] running on GCE node
I20260925 05:00:32.167937 31728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:32.169319 31728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:32.170564 31728 hybrid_clock.cc:648] HybridClock initialized: now 1790312432170527 us; error 49 us; skew 500 ppm
I20260925 05:00:32.170907 31728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:32 dist-test-slave-5n86 krb5kdc[31715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312432, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:32.173951 31728 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:00:32.175299 31728 webserver.cc:460] Webserver started at http://127.30.247.62:44963/ using document root <none> and password file <none>
I20260925 05:00:32.175917 31728 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:32.176023 31728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:32.176262 31728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:32.179190 31728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "b08d3efad63c472cb7c9e63cebe29c1e"
format_stamp: "Formatted at 2026-09-25 05:00:32 on dist-test-slave-5n86"
server_key: "f2b8a03a8cdd5e5bec4b92229601c167"
server_key_iv: "c5d11e530cf2d5c3c6f87b3835093986"
server_key_version: "encryptionkey@0"
I20260925 05:00:32.179896 31728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "b08d3efad63c472cb7c9e63cebe29c1e"
format_stamp: "Formatted at 2026-09-25 05:00:32 on dist-test-slave-5n86"
server_key: "f2b8a03a8cdd5e5bec4b92229601c167"
server_key_iv: "c5d11e530cf2d5c3c6f87b3835093986"
server_key_version: "encryptionkey@0"
I20260925 05:00:32.184930 31728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 05:00:32.188424 31743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:32.189953 31728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:00:32.190147 31728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "b08d3efad63c472cb7c9e63cebe29c1e"
format_stamp: "Formatted at 2026-09-25 05:00:32 on dist-test-slave-5n86"
server_key: "f2b8a03a8cdd5e5bec4b92229601c167"
server_key_iv: "c5d11e530cf2d5c3c6f87b3835093986"
server_key_version: "encryptionkey@0"
I20260925 05:00:32.190343 31728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:32.243369 31728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:32.247692 31728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:32.248009 31728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:32.266345 31728 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:37473
I20260925 05:00:32.266371 31794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:37473 every 8 connection(s)
I20260925 05:00:32.267710 31728 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:00:32.269218 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 31728
I20260925 05:00:32.269526 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:00:32.271325 31708 external_mini_cluster.cc:1428] Setting key d8928a10a6f77471c661b808bc2beb4d
I20260925 05:00:32.271654 31795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:32.282874 31795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b08d3efad63c472cb7c9e63cebe29c1e: Bootstrap starting.
I20260925 05:00:32.286005 31795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b08d3efad63c472cb7c9e63cebe29c1e: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:32.287017 31795 log.cc:824] T 00000000000000000000000000000000 P b08d3efad63c472cb7c9e63cebe29c1e: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:32.289639 31795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b08d3efad63c472cb7c9e63cebe29c1e: No bootstrap required, opened a new log
Sep 25 05:00:32 dist-test-slave-5n86 krb5kdc[31715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:00:32.296898 31795 raft_consensus.cc:348] T 00000000000000000000000000000000 P b08d3efad63c472cb7c9e63cebe29c1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08d3efad63c472cb7c9e63cebe29c1e" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37473 } }
I20260925 05:00:32.297266 31795 raft_consensus.cc:374] T 00000000000000000000000000000000 P b08d3efad63c472cb7c9e63cebe29c1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:32.297369 31795 raft_consensus.cc:729] T 00000000000000000000000000000000 P b08d3efad63c472cb7c9e63cebe29c1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b08d3efad63c472cb7c9e63cebe29c1e, State: Initialized, Role: FOLLOWER
I20260925 05:00:32.297963 31795 consensus_queue.cc:260] T 00000000000000000000000000000000 P b08d3efad63c472cb7c9e63cebe29c1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08d3efad63c472cb7c9e63cebe29c1e" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37473 } }
I20260925 05:00:32.298224 31795 raft_consensus.cc:388] T 00000000000000000000000000000000 P b08d3efad63c472cb7c9e63cebe29c1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:00:32.298378 31795 raft_consensus.cc:482] T 00000000000000000000000000000000 P b08d3efad63c472cb7c9e63cebe29c1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:00:32.298568 31795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b08d3efad63c472cb7c9e63cebe29c1e [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:32.301194 31795 raft_consensus.cc:504] T 00000000000000000000000000000000 P b08d3efad63c472cb7c9e63cebe29c1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08d3efad63c472cb7c9e63cebe29c1e" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37473 } }
I20260925 05:00:32.301687 31795 leader_election.cc:302] T 00000000000000000000000000000000 P b08d3efad63c472cb7c9e63cebe29c1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b08d3efad63c472cb7c9e63cebe29c1e; no voters:
I20260925 05:00:32.302248 31795 leader_election.cc:288] T 00000000000000000000000000000000 P b08d3efad63c472cb7c9e63cebe29c1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:00:32.302395 31800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b08d3efad63c472cb7c9e63cebe29c1e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:00:32.303771 31800 raft_consensus.cc:686] T 00000000000000000000000000000000 P b08d3efad63c472cb7c9e63cebe29c1e [term 1 LEADER]: Becoming Leader. State: Replica: b08d3efad63c472cb7c9e63cebe29c1e, State: Running, Role: LEADER
I20260925 05:00:32.304266 31800 consensus_queue.cc:237] T 00000000000000000000000000000000 P b08d3efad63c472cb7c9e63cebe29c1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08d3efad63c472cb7c9e63cebe29c1e" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37473 } }
I20260925 05:00:32.304422 31795 sys_catalog.cc:564] T 00000000000000000000000000000000 P b08d3efad63c472cb7c9e63cebe29c1e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:00:32.308175 31797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:32.275128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56158 (local address 127.30.247.62:37473)
0925 05:00:32.275929 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:32.275949 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:32.277052 (+ 1103us) server_negotiation.cc:407] Connection header received
0925 05:00:32.285391 (+ 8339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:32.285412 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:32.285809 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:32.286159 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:32.288929 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:32.290233 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:32.291795 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:32.292443 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:32.292683 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:32.298129 (+ 5446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:32.298137 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:32.298150 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:32.298217 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:32.301585 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:32.302625 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:32.302637 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:32.302652 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:32.302971 (+ 319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:32.303543 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:32.303551 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:32.303557 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:32.304281 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:32.304621 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:32.306020 (+ 1399us) server_negotiation.cc:299] Negotiation successful
0925 05:00:32.306532 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":360,"threads_started":1}
I20260925 05:00:32.308485 31801 sys_catalog.cc:455] T 00000000000000000000000000000000 P b08d3efad63c472cb7c9e63cebe29c1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b08d3efad63c472cb7c9e63cebe29c1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08d3efad63c472cb7c9e63cebe29c1e" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37473 } } }
I20260925 05:00:32.308889 31801 sys_catalog.cc:458] T 00000000000000000000000000000000 P b08d3efad63c472cb7c9e63cebe29c1e [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:32.309365 31802 sys_catalog.cc:455] T 00000000000000000000000000000000 P b08d3efad63c472cb7c9e63cebe29c1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b08d3efad63c472cb7c9e63cebe29c1e. Latest consensus state: current_term: 1 leader_uuid: "b08d3efad63c472cb7c9e63cebe29c1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08d3efad63c472cb7c9e63cebe29c1e" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37473 } } }
I20260925 05:00:32.309705 31802 sys_catalog.cc:458] T 00000000000000000000000000000000 P b08d3efad63c472cb7c9e63cebe29c1e [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:32.318634 31806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:00:32.323030 31806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:00:32.331259 31806 catalog_manager.cc:1347] Generated new cluster ID: b9271c9fbd6f4baca5da28707a1f5f1f
I20260925 05:00:32.331362 31806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:00:32.339972 31806 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:00:32.341773 31806 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:00:32.349488 31806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b08d3efad63c472cb7c9e63cebe29c1e: Generated new TSK 0
I20260925 05:00:32.350548 31806 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
I20260925 05:00:32.398372 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:0
--local_ip_for_outbound_sockets=127.30.247.1
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:37473
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:32.710245 31822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:32.710490 31822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:32.710712 31822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:32.717916 31822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:32.718027 31822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:32.718205 31822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:00:32.727522 31822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:37473
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:32.728711 31822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:32.729954 31822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:32.742465 31830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:32.742452 31828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:32.742745 31827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:32.743273 31822 server_base.cc:1034] running on GCE node
I20260925 05:00:32.743764 31822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:32.744616 31822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:32.745821 31822 hybrid_clock.cc:648] HybridClock initialized: now 1790312432745776 us; error 41 us; skew 500 ppm
I20260925 05:00:32.746133 31822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:32 dist-test-slave-5n86 krb5kdc[31715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312432, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:32.749147 31822 init.cc:377] Logged in from keytab as kudu/127.30.247.1@KRBTEST.COM (short username kudu)
I20260925 05:00:32.750538 31822 webserver.cc:460] Webserver started at http://127.30.247.1:35555/ using document root <none> and password file <none>
I20260925 05:00:32.751173 31822 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:32.751324 31822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:32.751642 31822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:32.754386 31822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-0/data/instance:
uuid: "5b49b58a8bbc4af088140f67d4b16c7b"
format_stamp: "Formatted at 2026-09-25 05:00:32 on dist-test-slave-5n86"
server_key: "b56a3a26e22067a1b57538b264976acf"
server_key_iv: "5839e248a6fbf88406a85111914559a4"
server_key_version: "encryptionkey@0"
I20260925 05:00:32.755110 31822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance:
uuid: "5b49b58a8bbc4af088140f67d4b16c7b"
format_stamp: "Formatted at 2026-09-25 05:00:32 on dist-test-slave-5n86"
server_key: "b56a3a26e22067a1b57538b264976acf"
server_key_iv: "5839e248a6fbf88406a85111914559a4"
server_key_version: "encryptionkey@0"
I20260925 05:00:32.759919 31822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 05:00:32.763320 31837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:32.764673 31822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:00:32.764834 31822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "5b49b58a8bbc4af088140f67d4b16c7b"
format_stamp: "Formatted at 2026-09-25 05:00:32 on dist-test-slave-5n86"
server_key: "b56a3a26e22067a1b57538b264976acf"
server_key_iv: "5839e248a6fbf88406a85111914559a4"
server_key_version: "encryptionkey@0"
I20260925 05:00:32.765005 31822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:32.790484 31822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:32.794567 31822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:32.794858 31822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:32.795894 31822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:00:32.797267 31822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:00:32.797339 31822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:32.797483 31822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:00:32.797569 31822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:32.831034 31822 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:42461
I20260925 05:00:32.831126 31949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:42461 every 8 connection(s)
I20260925 05:00:32.832340 31822 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:00:32.832891 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 31822
I20260925 05:00:32.833139 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance
I20260925 05:00:32.833946 31708 external_mini_cluster.cc:1428] Setting key 9f40100cc80a4d8b9f5f12984ebd40e5
Sep 25 05:00:32 dist-test-slave-5n86 krb5kdc[31715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312432, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.247.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.2@KRBTEST.COM" created.
I20260925 05:00:32.854130 31952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:32.835119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:42323 (local address 127.30.247.62:37473)
0925 05:00:32.835567 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:32.835573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:32.836825 (+ 1252us) server_negotiation.cc:407] Connection header received
0925 05:00:32.838623 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:32.838632 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:32.838720 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:32.838842 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:32.840283 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:32.841504 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:32.843076 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:32.843806 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:32.843947 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:32.847478 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:32.847487 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:32.847494 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:32.847550 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:32.850893 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:32.851590 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:32.851594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:32.851596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:32.851734 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:32.852223 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:32.852226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:32.852228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:32.852524 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:32.852721 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:32.853512 (+ 791us) server_negotiation.cc:299] Negotiation successful
0925 05:00:32.853729 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":216,"threads_started":1}
I20260925 05:00:32.855165 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:32.835781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37473 (local address 127.30.247.1:42323)
0925 05:00:32.836659 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:32.836711 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:32.838361 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:32.839031 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:32.839040 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:32.839548 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:32.840096 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:32.840113 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:32.841683 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:32.841688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:32.842928 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:32.842944 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:32.844023 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:32.844028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:32.844231 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:32.844936 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:32.844957 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:32.847259 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:32.851022 (+ 3763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:32.851029 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:32.851043 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:32.851475 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:32.851866 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:32.851874 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:32.851881 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:32.852094 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:32.852727 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:32.852734 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:32.853146 (+ 412us) client_negotiation.cc:769] Sending connection context
0925 05:00:32.853431 (+ 285us) client_negotiation.cc:240] Negotiation successful
0925 05:00:32.854000 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":359,"threads_started":1}
I20260925 05:00:32.856922 31950 heartbeater.cc:347] Connected to a master server at 127.30.247.62:37473
I20260925 05:00:32.857288 31950 heartbeater.cc:464] Registering TS with master...
I20260925 05:00:32.858192 31950 heartbeater.cc:511] Master 127.30.247.62:37473 requested a full tablet report, sending...
I20260925 05:00:32.860395 31760 ts_manager.cc:194] Registered new tserver with Master: 5b49b58a8bbc4af088140f67d4b16c7b (127.30.247.1:42461)
I20260925 05:00:32.861862 31760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.1@KRBTEST.COM'} at 127.30.247.1:42323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
I20260925 05:00:32.880501 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:0
--local_ip_for_outbound_sockets=127.30.247.2
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:37473
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:33.188159 31956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:33.188403 31956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:33.188512 31956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:33.195798 31956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:33.195885 31956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:33.196072 31956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:00:33.205351 31956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:37473
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:33.206461 31956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:33.207656 31956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:33.219859 31961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:33.219849 31962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:33.219944 31964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:33.220808 31956 server_base.cc:1034] running on GCE node
I20260925 05:00:33.221396 31956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:33.222242 31956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:33.223469 31956 hybrid_clock.cc:648] HybridClock initialized: now 1790312433223436 us; error 46 us; skew 500 ppm
I20260925 05:00:33.223779 31956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:33 dist-test-slave-5n86 krb5kdc[31715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312433, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:33.226655 31956 init.cc:377] Logged in from keytab as kudu/127.30.247.2@KRBTEST.COM (short username kudu)
I20260925 05:00:33.228010 31956 webserver.cc:460] Webserver started at http://127.30.247.2:35983/ using document root <none> and password file <none>
I20260925 05:00:33.228677 31956 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:33.228775 31956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:33.229014 31956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:33.231585 31956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-1/data/instance:
uuid: "156ef6f921814dc6a51ed445d66fab46"
format_stamp: "Formatted at 2026-09-25 05:00:33 on dist-test-slave-5n86"
server_key: "add3640b2e3e7772a19eeee6948e49f9"
server_key_iv: "a90f7d3b19fb957553c8d5b5ba12e72b"
server_key_version: "encryptionkey@0"
I20260925 05:00:33.232244 31956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance:
uuid: "156ef6f921814dc6a51ed445d66fab46"
format_stamp: "Formatted at 2026-09-25 05:00:33 on dist-test-slave-5n86"
server_key: "add3640b2e3e7772a19eeee6948e49f9"
server_key_iv: "a90f7d3b19fb957553c8d5b5ba12e72b"
server_key_version: "encryptionkey@0"
I20260925 05:00:33.236986 31956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 05:00:33.240190 31971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:33.241655 31956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:00:33.241827 31956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "156ef6f921814dc6a51ed445d66fab46"
format_stamp: "Formatted at 2026-09-25 05:00:33 on dist-test-slave-5n86"
server_key: "add3640b2e3e7772a19eeee6948e49f9"
server_key_iv: "a90f7d3b19fb957553c8d5b5ba12e72b"
server_key_version: "encryptionkey@0"
I20260925 05:00:33.242008 31956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:33.279100 31956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:33.283351 31956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:33.283638 31956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:33.284602 31956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:00:33.286000 31956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:00:33.286072 31956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:33.286235 31956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:00:33.286283 31956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:33.318204 31956 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:40741
I20260925 05:00:33.318287 32083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:40741 every 8 connection(s)
I20260925 05:00:33.319396 31956 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:00:33.325775 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 31956
I20260925 05:00:33.325958 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance
I20260925 05:00:33.326368 31708 external_mini_cluster.cc:1428] Setting key 87f94e2104145d588bb4c4ccbea463d3
Sep 25 05:00:33 dist-test-slave-5n86 krb5kdc[31715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312433, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:00:33.340001 31952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:33.321851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:42587 (local address 127.30.247.62:37473)
0925 05:00:33.322052 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:33.322059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:33.323229 (+ 1170us) server_negotiation.cc:407] Connection header received
0925 05:00:33.325212 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:33.325216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:33.325280 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:33.325371 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:33.326974 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:33.327883 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:33.329431 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:33.329823 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:33.329952 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:33.333410 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:33.333419 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:33.333428 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:33.333470 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:33.337070 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:33.337828 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:33.337835 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:33.337844 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:33.337968 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:33.338380 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:33.338385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:33.338390 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:33.338647 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:33.338810 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:33.339445 (+ 635us) server_negotiation.cc:299] Negotiation successful
0925 05:00:33.339669 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 05:00:33.340957 32086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:33.322376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37473 (local address 127.30.247.2:42587)
0925 05:00:33.323064 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:33.323109 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:33.324949 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:33.325593 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:33.325601 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:33.326213 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:33.326806 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:33.326824 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:33.328005 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:33.328011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:33.329291 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:33.329302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:33.330036 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:33.330043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:33.330264 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:33.330901 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:33.330926 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:33.333196 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:33.337222 (+ 4026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:33.337233 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:33.337249 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:33.337700 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:33.338075 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:33.338081 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:33.338084 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:33.338261 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:33.338785 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:33.338793 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:33.339091 (+ 298us) client_negotiation.cc:769] Sending connection context
0925 05:00:33.339346 (+ 255us) client_negotiation.cc:240] Negotiation successful
0925 05:00:33.339823 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":292,"threads_started":1}
WARNING: no policy specified for kudu/127.30.247.3@KRBTEST.COM; defaulting to no policy
I20260925 05:00:33.342443 32084 heartbeater.cc:347] Connected to a master server at 127.30.247.62:37473
I20260925 05:00:33.342774 32084 heartbeater.cc:464] Registering TS with master...
I20260925 05:00:33.343451 32084 heartbeater.cc:511] Master 127.30.247.62:37473 requested a full tablet report, sending...
I20260925 05:00:33.344810 31760 ts_manager.cc:194] Registered new tserver with Master: 156ef6f921814dc6a51ed445d66fab46 (127.30.247.2:40741)
I20260925 05:00:33.345647 31760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.2@KRBTEST.COM'} at 127.30.247.2:42587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
I20260925 05:00:33.371373 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:0
--local_ip_for_outbound_sockets=127.30.247.3
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:37473
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:33.675915 32089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:33.676165 32089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:33.676319 32089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:33.683202 32089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:33.683296 32089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:33.683512 32089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:00:33.692890 32089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:37473
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:33.694090 32089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:33.695393 32089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:33.707790 32094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:33.707748 32097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:33.707831 32095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:33.714035 32089 server_base.cc:1034] running on GCE node
I20260925 05:00:33.714573 32089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:33.715464 32089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:33.716635 32089 hybrid_clock.cc:648] HybridClock initialized: now 1790312433716595 us; error 47 us; skew 500 ppm
I20260925 05:00:33.716924 32089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:33 dist-test-slave-5n86 krb5kdc[31715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312433, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:33.719863 32089 init.cc:377] Logged in from keytab as kudu/127.30.247.3@KRBTEST.COM (short username kudu)
I20260925 05:00:33.721290 32089 webserver.cc:460] Webserver started at http://127.30.247.3:34437/ using document root <none> and password file <none>
I20260925 05:00:33.721911 32089 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:33.722003 32089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:33.722240 32089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:33.725044 32089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-2/data/instance:
uuid: "63ee6c39d93944cfa668d93a4cfa6b6a"
format_stamp: "Formatted at 2026-09-25 05:00:33 on dist-test-slave-5n86"
server_key: "c30272c17d64a8a87742885e83379fee"
server_key_iv: "68141faf13f8ae597df9d5d5bdabfdc4"
server_key_version: "encryptionkey@0"
I20260925 05:00:33.725713 32089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance:
uuid: "63ee6c39d93944cfa668d93a4cfa6b6a"
format_stamp: "Formatted at 2026-09-25 05:00:33 on dist-test-slave-5n86"
server_key: "c30272c17d64a8a87742885e83379fee"
server_key_iv: "68141faf13f8ae597df9d5d5bdabfdc4"
server_key_version: "encryptionkey@0"
I20260925 05:00:33.730412 32089 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 05:00:33.733489 32104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:33.734723 32089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:00:33.734840 32089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "63ee6c39d93944cfa668d93a4cfa6b6a"
format_stamp: "Formatted at 2026-09-25 05:00:33 on dist-test-slave-5n86"
server_key: "c30272c17d64a8a87742885e83379fee"
server_key_iv: "68141faf13f8ae597df9d5d5bdabfdc4"
server_key_version: "encryptionkey@0"
I20260925 05:00:33.734992 32089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:33.756836 32089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:33.760823 32089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:33.761098 32089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:33.761951 32089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:00:33.763134 32089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:00:33.763204 32089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:33.763283 32089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:00:33.763325 32089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:33.795785 32089 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:35755
I20260925 05:00:33.795868 32216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:35755 every 8 connection(s)
I20260925 05:00:33.797061 32089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:00:33.805483 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 32089
I20260925 05:00:33.805676 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance
I20260925 05:00:33.806082 31708 external_mini_cluster.cc:1428] Setting key e92858eb574e82825d68a274a91db5c4
Sep 25 05:00:33 dist-test-slave-5n86 krb5kdc[31715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312433, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:00:33.816048 31952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:33.799330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:41833 (local address 127.30.247.62:37473)
0925 05:00:33.799518 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:33.799522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:33.800730 (+ 1208us) server_negotiation.cc:407] Connection header received
0925 05:00:33.802526 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:33.802531 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:33.802622 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:33.802730 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:33.804155 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:33.805039 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:33.806779 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:33.807397 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:33.807522 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:33.811163 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:33.811168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:33.811172 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:33.811201 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:33.813534 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:33.814177 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:33.814181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:33.814183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:33.814283 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:33.814641 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:33.814644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:33.814646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:33.814888 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:33.815001 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:33.815675 (+ 674us) server_negotiation.cc:299] Negotiation successful
0925 05:00:33.815820 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 05:00:33.817025 32219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:33.799818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37473 (local address 127.30.247.3:41833)
0925 05:00:33.800501 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:33.800613 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:33.802259 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:33.802956 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:33.802964 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:33.803441 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:33.803959 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:33.803975 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:33.805178 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:33.805184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:33.806643 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:33.806654 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:33.807787 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:33.807795 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:33.808111 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:33.808841 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:33.808862 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:33.810978 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:33.813664 (+ 2686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:33.813670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:33.813681 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:33.814076 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:33.814370 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:33.814374 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:33.814376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:33.814546 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:33.815030 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:33.815057 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:33.815382 (+ 325us) client_negotiation.cc:769] Sending connection context
0925 05:00:33.815594 (+ 212us) client_negotiation.cc:240] Negotiation successful
0925 05:00:33.815927 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":267,"threads_started":1}
I20260925 05:00:33.818233 32217 heartbeater.cc:347] Connected to a master server at 127.30.247.62:37473
I20260925 05:00:33.818508 32217 heartbeater.cc:464] Registering TS with master...
I20260925 05:00:33.819147 32217 heartbeater.cc:511] Master 127.30.247.62:37473 requested a full tablet report, sending...
I20260925 05:00:33.820482 31760 ts_manager.cc:194] Registered new tserver with Master: 63ee6c39d93944cfa668d93a4cfa6b6a (127.30.247.3:35755)
I20260925 05:00:33.821436 31760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.3@KRBTEST.COM'} at 127.30.247.3:41833
I20260925 05:00:33.821926 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 05:00:33 dist-test-slave-5n86 krb5kdc[31715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312433, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 05:00:33 dist-test-slave-5n86 krb5kdc[31715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312433, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.247.1@KRBTEST.COM
I20260925 05:00:33.864802 32224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:33.850348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54860 (local address 127.30.247.1:42461)
0925 05:00:33.850977 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:33.850984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:33.851010 (+ 26us) server_negotiation.cc:407] Connection header received
0925 05:00:33.851070 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:33.851076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:33.851288 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:33.851424 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:33.852873 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:33.853865 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:33.855141 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:33.855719 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:33.855827 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:33.859564 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:33.859575 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:33.859593 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:33.859628 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:33.862479 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:33.863043 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:33.863049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:33.863052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:33.863175 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:33.863492 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:33.863495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:33.863497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:33.863805 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:33.864021 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:33.864247 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 05:00:33.864431 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":357,"threads_started":1}
I20260925 05:00:33.865574 31950 heartbeater.cc:503] Master 127.30.247.62:37473 was elected leader, sending a full tablet report...
W20260925 05:00:33.866113 31865 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54860
Sep 25 05:00:33 dist-test-slave-5n86 krb5kdc[31715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 05:00:33 dist-test-slave-5n86 krb5kdc[31715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.1@KRBTEST.COM
I20260925 05:00:33.908691 32224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:33.895313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54862 (local address 127.30.247.1:42461)
0925 05:00:33.895558 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:33.895567 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:33.895850 (+ 283us) server_negotiation.cc:407] Connection header received
0925 05:00:33.896067 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:33.896071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:33.896162 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:33.896270 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:33.897627 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:33.898325 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:33.899523 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:33.899927 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:33.900075 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:33.903680 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:33.903685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:33.903688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:33.903719 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:33.906202 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:33.906823 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:33.906834 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:33.906838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:33.906948 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:33.907358 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:33.907363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:33.907367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:33.907606 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:33.907837 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:33.908137 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 05:00:33.908345 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 05:00:33.910033 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 31822
I20260925 05:00:33.921670 32224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:33.910960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54872 (local address 127.30.247.1:42461)
0925 05:00:33.911120 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:33.911124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:33.911381 (+ 257us) server_negotiation.cc:407] Connection header received
0925 05:00:33.911492 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:33.911495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:33.911559 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:33.911633 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:33.912700 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:33.913326 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:33.914458 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:33.914860 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:33.914947 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:33.917081 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:33.917085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:33.917088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:33.917114 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:33.919485 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:33.920078 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:33.920082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:33.920084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:33.920217 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:33.920506 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:33.920509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:33.920511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:33.920824 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:33.921002 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:33.921218 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 05:00:33.921350 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 05:00:33.922355 31945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:00:34.008031 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 31822
I20260925 05:00:34.030536 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 31956
Sep 25 05:00:34 dist-test-slave-5n86 krb5kdc[31715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.2@KRBTEST.COM
I20260925 05:00:34.046303 32231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:34.031405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39744 (local address 127.30.247.2:40741)
0925 05:00:34.031944 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:34.031951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:34.031975 (+ 24us) server_negotiation.cc:407] Connection header received
0925 05:00:34.032044 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:34.032051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:34.032251 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:34.032380 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:34.033538 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:34.034537 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:34.035772 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:34.036318 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:34.036417 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:34.040852 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:34.040863 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:34.040876 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:34.040913 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:34.043496 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:34.044121 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:34.044128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:34.044131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:34.044247 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:34.044630 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:34.044637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:34.044640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:34.044955 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:34.045245 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:34.045544 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 05:00:34.045826 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":293,"threads_started":1}
I20260925 05:00:34.047485 32079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:00:34.128790 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 31956
I20260925 05:00:34.145051 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 32089
Sep 25 05:00:34 dist-test-slave-5n86 krb5kdc[31715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.3@KRBTEST.COM
I20260925 05:00:34.161281 32233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:34.145987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32966 (local address 127.30.247.3:35755)
0925 05:00:34.146573 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:34.146580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:34.146608 (+ 28us) server_negotiation.cc:407] Connection header received
0925 05:00:34.146666 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:34.146672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:34.146882 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:34.147053 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:34.148318 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:34.149420 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:34.150733 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:34.151285 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:34.151426 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:34.155580 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:34.155590 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:34.155604 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:34.155641 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:34.158423 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:34.159091 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:34.159100 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:34.159102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:34.159225 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:34.159579 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:34.159585 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:34.159588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:34.159904 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:34.160232 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:34.160514 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 05:00:34.160978 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":309,"threads_started":1}
I20260925 05:00:34.162760 32212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:00:34.251324 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 32089
I20260925 05:00:34.267021 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 31728
I20260925 05:00:34.268188 31790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:00:34.369964 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 31728
2026-09-25T05:00:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3858 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:00:34 dist-test-slave-5n86 krb5kdc[32238](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:00:34 dist-test-slave-5n86 krb5kdc[32238](info): set up 2 sockets
Sep 25 05:00:34 dist-test-slave-5n86 krb5kdc[32238](info): commencing operation
krb5kdc: starting...
W20260925 05:00:35.440624 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:00:35 dist-test-slave-5n86 krb5kdc[32238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:00:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:00:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:00:35.612337 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:35835
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:45625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:35835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:35.917857 32251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:35.918186 32251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:35.918336 32251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:35.925264 32251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:00:35.925401 32251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:35.925482 32251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:35.925582 32251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:00:35.925653 32251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:00:35.934954 32251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:45625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:35835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:35835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:35.936234 32251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:35.937534 32251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:35.948975 32257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:35.948984 32256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:35.949458 32259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:35.949921 32251 server_base.cc:1034] running on GCE node
I20260925 05:00:35.950573 32251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:35.951889 32251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:35.953147 32251 hybrid_clock.cc:648] HybridClock initialized: now 1790312435953110 us; error 62 us; skew 500 ppm
I20260925 05:00:35.953532 32251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:35 dist-test-slave-5n86 krb5kdc[32238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312435, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:35.965312 32251 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:00:35.967376 32251 webserver.cc:460] Webserver started at http://127.30.247.62:39795/ using document root <none> and password file <none>
I20260925 05:00:35.968359 32251 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:35.968549 32251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:35.968986 32251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:35.972052 32251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "cba0d7e01cb245ceb478df36ca5fce4b"
format_stamp: "Formatted at 2026-09-25 05:00:35 on dist-test-slave-5n86"
server_key: "edede846cfbacd051ea11bdfbf3fef0b"
server_key_iv: "66de101afda1036eff02eace457dbb22"
server_key_version: "encryptionkey@0"
I20260925 05:00:35.972839 32251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "cba0d7e01cb245ceb478df36ca5fce4b"
format_stamp: "Formatted at 2026-09-25 05:00:35 on dist-test-slave-5n86"
server_key: "edede846cfbacd051ea11bdfbf3fef0b"
server_key_iv: "66de101afda1036eff02eace457dbb22"
server_key_version: "encryptionkey@0"
I20260925 05:00:35.977732 32251 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 05:00:35.980978 32266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:35.982231 32251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 05:00:35.982409 32251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "cba0d7e01cb245ceb478df36ca5fce4b"
format_stamp: "Formatted at 2026-09-25 05:00:35 on dist-test-slave-5n86"
server_key: "edede846cfbacd051ea11bdfbf3fef0b"
server_key_iv: "66de101afda1036eff02eace457dbb22"
server_key_version: "encryptionkey@0"
I20260925 05:00:35.982589 32251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:36.002323 32251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:36.006394 32251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:36.006685 32251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:36.025149 32251 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:35835
I20260925 05:00:36.025156 32317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:35835 every 8 connection(s)
I20260925 05:00:36.026469 32251 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:00:36.027586 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 32251
I20260925 05:00:36.027822 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:00:36.028447 31708 external_mini_cluster.cc:1428] Setting key c7c7c26ce590e72f348b31f59515c521
I20260925 05:00:36.030666 32318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 05:00:36 dist-test-slave-5n86 krb5kdc[32238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:00:36.043572 32318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cba0d7e01cb245ceb478df36ca5fce4b: Bootstrap starting.
I20260925 05:00:36.047608 32318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cba0d7e01cb245ceb478df36ca5fce4b: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:36.048906 32318 log.cc:824] T 00000000000000000000000000000000 P cba0d7e01cb245ceb478df36ca5fce4b: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:36.052166 32318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cba0d7e01cb245ceb478df36ca5fce4b: No bootstrap required, opened a new log
I20260925 05:00:36.052433 32321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:36.030439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58688 (local address 127.30.247.62:35835)
0925 05:00:36.031205 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:36.031224 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:36.031270 (+ 46us) server_negotiation.cc:407] Connection header received
0925 05:00:36.033468 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:36.033494 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:36.033986 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:36.034425 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:36.035676 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:36.037786 (+ 2110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:36.038962 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:36.039958 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:36.040148 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:36.044153 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:36.044162 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:36.044179 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:36.044228 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:36.048069 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:36.048804 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:36.048813 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:36.048821 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:36.049028 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:36.049382 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:36.049390 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:36.049396 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:36.049991 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:36.050272 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:36.050710 (+ 438us) server_negotiation.cc:299] Negotiation successful
0925 05:00:36.051127 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":308,"threads_started":1}
I20260925 05:00:36.060194 32318 raft_consensus.cc:348] T 00000000000000000000000000000000 P cba0d7e01cb245ceb478df36ca5fce4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cba0d7e01cb245ceb478df36ca5fce4b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35835 } }
I20260925 05:00:36.060577 32318 raft_consensus.cc:374] T 00000000000000000000000000000000 P cba0d7e01cb245ceb478df36ca5fce4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:36.060672 32318 raft_consensus.cc:729] T 00000000000000000000000000000000 P cba0d7e01cb245ceb478df36ca5fce4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cba0d7e01cb245ceb478df36ca5fce4b, State: Initialized, Role: FOLLOWER
I20260925 05:00:36.061226 32318 consensus_queue.cc:260] T 00000000000000000000000000000000 P cba0d7e01cb245ceb478df36ca5fce4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cba0d7e01cb245ceb478df36ca5fce4b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35835 } }
I20260925 05:00:36.061426 32318 raft_consensus.cc:388] T 00000000000000000000000000000000 P cba0d7e01cb245ceb478df36ca5fce4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:00:36.061522 32318 raft_consensus.cc:482] T 00000000000000000000000000000000 P cba0d7e01cb245ceb478df36ca5fce4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:00:36.061645 32318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cba0d7e01cb245ceb478df36ca5fce4b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:36.063683 32318 raft_consensus.cc:504] T 00000000000000000000000000000000 P cba0d7e01cb245ceb478df36ca5fce4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cba0d7e01cb245ceb478df36ca5fce4b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35835 } }
I20260925 05:00:36.064123 32318 leader_election.cc:302] T 00000000000000000000000000000000 P cba0d7e01cb245ceb478df36ca5fce4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cba0d7e01cb245ceb478df36ca5fce4b; no voters:
I20260925 05:00:36.064777 32318 leader_election.cc:288] T 00000000000000000000000000000000 P cba0d7e01cb245ceb478df36ca5fce4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:00:36.064932 32323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cba0d7e01cb245ceb478df36ca5fce4b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:00:36.065583 32323 raft_consensus.cc:686] T 00000000000000000000000000000000 P cba0d7e01cb245ceb478df36ca5fce4b [term 1 LEADER]: Becoming Leader. State: Replica: cba0d7e01cb245ceb478df36ca5fce4b, State: Running, Role: LEADER
I20260925 05:00:36.066165 32323 consensus_queue.cc:237] T 00000000000000000000000000000000 P cba0d7e01cb245ceb478df36ca5fce4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cba0d7e01cb245ceb478df36ca5fce4b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35835 } }
I20260925 05:00:36.066988 32318 sys_catalog.cc:564] T 00000000000000000000000000000000 P cba0d7e01cb245ceb478df36ca5fce4b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:00:36.069231 32325 sys_catalog.cc:455] T 00000000000000000000000000000000 P cba0d7e01cb245ceb478df36ca5fce4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cba0d7e01cb245ceb478df36ca5fce4b. Latest consensus state: current_term: 1 leader_uuid: "cba0d7e01cb245ceb478df36ca5fce4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cba0d7e01cb245ceb478df36ca5fce4b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35835 } } }
I20260925 05:00:36.069690 32325 sys_catalog.cc:458] T 00000000000000000000000000000000 P cba0d7e01cb245ceb478df36ca5fce4b [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:36.070256 32324 sys_catalog.cc:455] T 00000000000000000000000000000000 P cba0d7e01cb245ceb478df36ca5fce4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cba0d7e01cb245ceb478df36ca5fce4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cba0d7e01cb245ceb478df36ca5fce4b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35835 } } }
I20260925 05:00:36.070518 32324 sys_catalog.cc:458] T 00000000000000000000000000000000 P cba0d7e01cb245ceb478df36ca5fce4b [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:36.072468 32332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:00:36.077297 32332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:00:36.086313 32332 catalog_manager.cc:1347] Generated new cluster ID: e5a0ddf986a842febeafe634f378f7e9
I20260925 05:00:36.086421 32332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:00:36.098914 32332 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:00:36.100778 32332 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:00:36.108143 32332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cba0d7e01cb245ceb478df36ca5fce4b: Generated new TSK 0
I20260925 05:00:36.108975 32332 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
I20260925 05:00:36.163746 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:0
--local_ip_for_outbound_sockets=127.30.247.1
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:35835
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:45625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:36.491597 32344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:36.491876 32344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:36.492034 32344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:36.499342 32344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:36.499456 32344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:36.499631 32344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:00:36.509276 32344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:45625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:35835
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:36.510670 32344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:36.511960 32344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:36.524714 32349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:36.524677 32352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:36.524745 32350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:36.525830 32344 server_base.cc:1034] running on GCE node
I20260925 05:00:36.526401 32344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:36.527280 32344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:36.528587 32344 hybrid_clock.cc:648] HybridClock initialized: now 1790312436528490 us; error 60 us; skew 500 ppm
I20260925 05:00:36.528913 32344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:36 dist-test-slave-5n86 krb5kdc[32238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312436, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:36.531838 32344 init.cc:377] Logged in from keytab as kudu/127.30.247.1@KRBTEST.COM (short username kudu)
I20260925 05:00:36.533350 32344 webserver.cc:460] Webserver started at http://127.30.247.1:34567/ using document root <none> and password file <none>
I20260925 05:00:36.534006 32344 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:36.534152 32344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:36.534448 32344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:36.537240 32344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-0/data/instance:
uuid: "b26c32c5e378412cad65fe4ac2abe998"
format_stamp: "Formatted at 2026-09-25 05:00:36 on dist-test-slave-5n86"
server_key: "f10bb71a2345601ef71519419996a5ef"
server_key_iv: "d675c2c31548bdd0301c3902750cef08"
server_key_version: "encryptionkey@0"
I20260925 05:00:36.537985 32344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance:
uuid: "b26c32c5e378412cad65fe4ac2abe998"
format_stamp: "Formatted at 2026-09-25 05:00:36 on dist-test-slave-5n86"
server_key: "f10bb71a2345601ef71519419996a5ef"
server_key_iv: "d675c2c31548bdd0301c3902750cef08"
server_key_version: "encryptionkey@0"
I20260925 05:00:36.542730 32344 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 05:00:36.545933 32359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:36.547127 32344 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 05:00:36.547295 32344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "b26c32c5e378412cad65fe4ac2abe998"
format_stamp: "Formatted at 2026-09-25 05:00:36 on dist-test-slave-5n86"
server_key: "f10bb71a2345601ef71519419996a5ef"
server_key_iv: "d675c2c31548bdd0301c3902750cef08"
server_key_version: "encryptionkey@0"
I20260925 05:00:36.547475 32344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:36.561688 32344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:36.565379 32344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:36.565651 32344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:36.566511 32344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:00:36.567839 32344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:00:36.567936 32344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:36.568100 32344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:00:36.568181 32344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:36.600818 32344 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:46497
I20260925 05:00:36.600867 32471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:46497 every 8 connection(s)
I20260925 05:00:36.602065 32344 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:00:36.611555 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 32344
I20260925 05:00:36.611773 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance
I20260925 05:00:36.612505 31708 external_mini_cluster.cc:1428] Setting key db219d30096f4a34dd3f336bb3bc8fc5
Sep 25 05:00:36 dist-test-slave-5n86 krb5kdc[32238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312436, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:00:36.623952 32474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:36.604685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:33215 (local address 127.30.247.62:35835)
0925 05:00:36.605129 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:36.605134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:36.605920 (+ 786us) server_negotiation.cc:407] Connection header received
0925 05:00:36.607936 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:36.607941 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:36.608040 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:36.608161 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:36.609681 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:36.610486 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:36.612657 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:36.613059 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:36.613168 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:36.617067 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:36.617076 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:36.617084 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:36.617130 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:36.620608 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:36.621580 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:36.621585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:36.621587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:36.621718 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:36.622263 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:36.622267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:36.622269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:36.622538 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:36.622646 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:36.623549 (+ 903us) server_negotiation.cc:299] Negotiation successful
0925 05:00:36.623703 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":243,"threads_started":1}
I20260925 05:00:36.625497 32475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:36.605006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:35835 (local address 127.30.247.1:33215)
0925 05:00:36.605751 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:36.605803 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:36.607622 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:36.608412 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:36.608420 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:36.608970 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:36.609516 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:36.609533 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:36.610637 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:36.610641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:36.612475 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:36.612504 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:36.613285 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:36.613292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:36.613545 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:36.614218 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:36.614239 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:36.616828 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:36.620741 (+ 3913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:36.620753 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:36.620771 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:36.621416 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:36.621832 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:36.621839 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:36.621846 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:36.622105 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:36.622663 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:36.622674 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:36.623123 (+ 449us) client_negotiation.cc:769] Sending connection context
0925 05:00:36.623432 (+ 309us) client_negotiation.cc:240] Negotiation successful
0925 05:00:36.623978 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":283,"threads_started":1}
I20260925 05:00:36.627044 32472 heartbeater.cc:347] Connected to a master server at 127.30.247.62:35835
I20260925 05:00:36.627416 32472 heartbeater.cc:464] Registering TS with master...
I20260925 05:00:36.628367 32472 heartbeater.cc:511] Master 127.30.247.62:35835 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.30.247.2@KRBTEST.COM; defaulting to no policy
I20260925 05:00:36.630613 32283 ts_manager.cc:194] Registered new tserver with Master: b26c32c5e378412cad65fe4ac2abe998 (127.30.247.1:46497)
I20260925 05:00:36.632025 32283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.1@KRBTEST.COM'} at 127.30.247.1:33215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
I20260925 05:00:36.657032 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:0
--local_ip_for_outbound_sockets=127.30.247.2
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:35835
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:45625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:36.979774 32478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:36.980186 32478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:36.980360 32478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:36.989331 32478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:36.989434 32478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:36.989607 32478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:00:36.999440 32478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:45625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:35835
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:37.000778 32478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:37.002079 32478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:37.015313 32483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:37.015285 32486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:37.015235 32484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:37.016085 32478 server_base.cc:1034] running on GCE node
I20260925 05:00:37.016772 32478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:37.017668 32478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:37.018918 32478 hybrid_clock.cc:648] HybridClock initialized: now 1790312437018879 us; error 55 us; skew 500 ppm
I20260925 05:00:37.019352 32478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:37 dist-test-slave-5n86 krb5kdc[32238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312437, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:37.023038 32478 init.cc:377] Logged in from keytab as kudu/127.30.247.2@KRBTEST.COM (short username kudu)
I20260925 05:00:37.024923 32478 webserver.cc:460] Webserver started at http://127.30.247.2:34803/ using document root <none> and password file <none>
I20260925 05:00:37.025700 32478 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:37.025826 32478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:37.026119 32478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:37.028934 32478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-1/data/instance:
uuid: "93448f97416746559e65ed42e9e65d64"
format_stamp: "Formatted at 2026-09-25 05:00:37 on dist-test-slave-5n86"
server_key: "c10a23cb3d2d8fb2ae351f0d7f073c0c"
server_key_iv: "64653ebbe4139177838ea9fc64d45a10"
server_key_version: "encryptionkey@0"
I20260925 05:00:37.029690 32478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance:
uuid: "93448f97416746559e65ed42e9e65d64"
format_stamp: "Formatted at 2026-09-25 05:00:37 on dist-test-slave-5n86"
server_key: "c10a23cb3d2d8fb2ae351f0d7f073c0c"
server_key_iv: "64653ebbe4139177838ea9fc64d45a10"
server_key_version: "encryptionkey@0"
I20260925 05:00:37.034731 32478 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:00:37.038117 32493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:37.039423 32478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:00:37.039587 32478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "93448f97416746559e65ed42e9e65d64"
format_stamp: "Formatted at 2026-09-25 05:00:37 on dist-test-slave-5n86"
server_key: "c10a23cb3d2d8fb2ae351f0d7f073c0c"
server_key_iv: "64653ebbe4139177838ea9fc64d45a10"
server_key_version: "encryptionkey@0"
I20260925 05:00:37.039778 32478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:37.065335 32478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:37.069654 32478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:37.069933 32478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:37.070974 32478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:00:37.072300 32478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:00:37.072397 32478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:37.072554 32478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:00:37.072625 32478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260925 05:00:37.106149 32478 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:36603
I20260925 05:00:37.106244 32605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:36603 every 8 connection(s)
I20260925 05:00:37.107419 32478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:00:37.112923 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 32478
I20260925 05:00:37.113121 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance
I20260925 05:00:37.113673 31708 external_mini_cluster.cc:1428] Setting key eb2009e11707a598841f3527552d1626
Sep 25 05:00:37 dist-test-slave-5n86 krb5kdc[32238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312437, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:00:37.129161 32474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:37.110109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:39597 (local address 127.30.247.62:35835)
0925 05:00:37.110288 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:37.110293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:37.111436 (+ 1143us) server_negotiation.cc:407] Connection header received
0925 05:00:37.113412 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:37.113419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:37.113509 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:37.113633 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:37.115408 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.116120 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.117742 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.118151 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.118295 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:37.121926 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:37.121936 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:37.121943 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:37.121988 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:37.125711 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:37.126537 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:37.126543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:37.126548 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:37.126712 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:37.127198 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:37.127206 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:37.127221 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:37.127623 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:37.127792 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:37.128459 (+ 667us) server_negotiation.cc:299] Negotiation successful
0925 05:00:37.128746 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:00:37.129976 32608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:37.110603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:35835 (local address 127.30.247.2:39597)
0925 05:00:37.111268 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:37.111310 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:37.113098 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:37.114005 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:37.114013 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:37.114652 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:37.115235 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:37.115253 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.116275 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.116282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:37.117583 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:37.117602 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.118383 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.118390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:37.118624 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:37.119385 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:37.119415 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:37.121696 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:37.125857 (+ 4161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:37.125864 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:37.125877 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:37.126395 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:37.126848 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:37.126852 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:37.126854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:37.127061 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:37.127762 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:37.127767 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:37.128083 (+ 316us) client_negotiation.cc:769] Sending connection context
0925 05:00:37.128308 (+ 225us) client_negotiation.cc:240] Negotiation successful
0925 05:00:37.128796 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":256,"threads_started":1}
WARNING: no policy specified for kudu/127.30.247.3@KRBTEST.COM; defaulting to no policy
I20260925 05:00:37.131620 32606 heartbeater.cc:347] Connected to a master server at 127.30.247.62:35835
I20260925 05:00:37.131944 32606 heartbeater.cc:464] Registering TS with master...
I20260925 05:00:37.132722 32606 heartbeater.cc:511] Master 127.30.247.62:35835 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.3@KRBTEST.COM" created.
I20260925 05:00:37.134337 32283 ts_manager.cc:194] Registered new tserver with Master: 93448f97416746559e65ed42e9e65d64 (127.30.247.2:36603)
I20260925 05:00:37.135612 32283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.2@KRBTEST.COM'} at 127.30.247.2:39597
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
I20260925 05:00:37.155947 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:0
--local_ip_for_outbound_sockets=127.30.247.3
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:35835
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:45625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:37.468984 32611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:37.469332 32611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:37.469430 32611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:37.476408 32611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:37.476503 32611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:37.476686 32611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:00:37.486245 32611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:45625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:35835
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:37.487514 32611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:37.488912 32611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:37.501879 32617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:37.501840 32619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:37.501902 32616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:37.502676 32611 server_base.cc:1034] running on GCE node
I20260925 05:00:37.503300 32611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:37.504251 32611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:37.505481 32611 hybrid_clock.cc:648] HybridClock initialized: now 1790312437505423 us; error 46 us; skew 500 ppm
I20260925 05:00:37.505796 32611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:37 dist-test-slave-5n86 krb5kdc[32238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312437, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:37.508919 32611 init.cc:377] Logged in from keytab as kudu/127.30.247.3@KRBTEST.COM (short username kudu)
I20260925 05:00:37.510350 32611 webserver.cc:460] Webserver started at http://127.30.247.3:37653/ using document root <none> and password file <none>
I20260925 05:00:37.511035 32611 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:37.511178 32611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:37.511557 32611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:37.514394 32611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-2/data/instance:
uuid: "f88db44e0cab4f2b8a74c00624dac94f"
format_stamp: "Formatted at 2026-09-25 05:00:37 on dist-test-slave-5n86"
server_key: "264589f28ba0148562c9b55f20331481"
server_key_iv: "40c10e856e0144693df58b7950319eed"
server_key_version: "encryptionkey@0"
I20260925 05:00:37.515100 32611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance:
uuid: "f88db44e0cab4f2b8a74c00624dac94f"
format_stamp: "Formatted at 2026-09-25 05:00:37 on dist-test-slave-5n86"
server_key: "264589f28ba0148562c9b55f20331481"
server_key_iv: "40c10e856e0144693df58b7950319eed"
server_key_version: "encryptionkey@0"
I20260925 05:00:37.520165 32611 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 05:00:37.523835 32626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:37.525293 32611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:00:37.525431 32611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "f88db44e0cab4f2b8a74c00624dac94f"
format_stamp: "Formatted at 2026-09-25 05:00:37 on dist-test-slave-5n86"
server_key: "264589f28ba0148562c9b55f20331481"
server_key_iv: "40c10e856e0144693df58b7950319eed"
server_key_version: "encryptionkey@0"
I20260925 05:00:37.525601 32611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:37.563491 32611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:37.567869 32611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:37.568167 32611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:37.569190 32611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:00:37.570547 32611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:00:37.570616 32611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:37.570700 32611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:00:37.570745 32611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:37.602829 32611 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:42953
I20260925 05:00:37.602924 32738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:42953 every 8 connection(s)
I20260925 05:00:37.604161 32611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:00:37.611645 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 32611
I20260925 05:00:37.611848 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance
I20260925 05:00:37.612386 31708 external_mini_cluster.cc:1428] Setting key 0c6fa3d8a18a3eaf48e39f750a193eab
Sep 25 05:00:37 dist-test-slave-5n86 krb5kdc[32238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312437, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:00:37.624264 32474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:37.606828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:55269 (local address 127.30.247.62:35835)
0925 05:00:37.607009 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:37.607013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:37.607952 (+ 939us) server_negotiation.cc:407] Connection header received
0925 05:00:37.609829 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:37.609837 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:37.609971 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:37.610085 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:37.611574 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.612749 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.614740 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.615251 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.615387 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:37.618767 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:37.618772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:37.618775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:37.618806 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:37.621346 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:37.622069 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:37.622076 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:37.622079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:37.622209 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:37.622622 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:37.622629 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:37.622632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:37.622894 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:37.623049 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:37.623708 (+ 659us) server_negotiation.cc:299] Negotiation successful
0925 05:00:37.623913 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 05:00:37.625273 32741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:37.607111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:35835 (local address 127.30.247.3:55269)
0925 05:00:37.607777 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:37.607820 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:37.609571 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:37.610291 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:37.610300 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:37.610789 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:37.611371 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:37.611388 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.612907 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.612916 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:37.614606 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:37.614620 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.615485 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.615493 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:37.615724 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:37.616383 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:37.616405 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:37.618565 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:37.621491 (+ 2926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:37.621500 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:37.621518 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:37.621932 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:37.622329 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:37.622333 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:37.622337 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:37.622512 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:37.623040 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:37.623047 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:37.623360 (+ 313us) client_negotiation.cc:769] Sending connection context
0925 05:00:37.623622 (+ 262us) client_negotiation.cc:240] Negotiation successful
0925 05:00:37.624145 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":265,"threads_started":1}
I20260925 05:00:37.626876 32739 heartbeater.cc:347] Connected to a master server at 127.30.247.62:35835
I20260925 05:00:37.627228 32739 heartbeater.cc:464] Registering TS with master...
I20260925 05:00:37.628018 32739 heartbeater.cc:511] Master 127.30.247.62:35835 requested a full tablet report, sending...
I20260925 05:00:37.629519 32283 ts_manager.cc:194] Registered new tserver with Master: f88db44e0cab4f2b8a74c00624dac94f (127.30.247.3:42953)
I20260925 05:00:37.630362 32283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.3@KRBTEST.COM'} at 127.30.247.3:55269
I20260925 05:00:37.634969 32472 heartbeater.cc:503] Master 127.30.247.62:35835 was elected leader, sending a full tablet report...
I20260925 05:00:37.639985 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:00:37.657819 32474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:37.646189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58702 (local address 127.30.247.62:35835)
0925 05:00:37.646383 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:37.646392 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:37.646852 (+ 460us) server_negotiation.cc:407] Connection header received
0925 05:00:37.646970 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:37.646974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:37.647032 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:37.647120 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:37.648459 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.649241 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.650567 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.650979 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.651091 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:37.653313 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:37.653318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:37.653321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:37.653348 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:37.655593 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:37.656134 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:37.656138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:37.656140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:37.656253 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:37.656583 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:37.656588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:37.656591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:37.656896 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:37.657064 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:37.657332 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 05:00:37.657477 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:00:37.663686 32283 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:00:37.670557 32283 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:00:37.707192 32755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:37.681787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:42953 (local address 127.0.0.1:57360)
0925 05:00:37.697684 (+ 15897us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:37.697720 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:37.697895 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:37.700070 (+ 2175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:37.700080 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:37.700105 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:00:37.700643 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:37.700659 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.702354 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.702359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:37.704126 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:37.704138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.706107 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.706113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:37.706293 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:37.706354 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 05:00:37.706496 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 05:00:37.706695 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15654,"mutex_wait_us":53,"thread_start_us":14766,"threads_started":1}
I20260925 05:00:37.707317 32754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:37.682197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44504 (local address 127.30.247.1:46497)
0925 05:00:37.699598 (+ 17401us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:37.699607 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:37.699632 (+ 25us) server_negotiation.cc:407] Connection header received
0925 05:00:37.699718 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:37.699725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:37.699926 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:37.700116 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:00:37.700989 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.702496 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.705078 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.706274 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.706423 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:37.706643 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:37.706865 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 05:00:37.707012 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17247,"thread_start_us":17100,"threads_started":1}
I20260925 05:00:37.707365 32751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:37.681969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:46497 (local address 127.0.0.1:44504)
0925 05:00:37.697780 (+ 15811us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:37.697805 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:37.697958 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:37.700203 (+ 2245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:37.700208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:37.700240 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:00:37.700800 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:37.700813 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.702642 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.702647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:37.704907 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:37.704919 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.706489 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.706495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:37.706675 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:37.706713 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 05:00:37.706807 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 05:00:37.706937 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15641,"mutex_wait_us":2,"thread_start_us":16593,"threads_started":1}
I20260925 05:00:37.708176 32753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:37.682000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57360 (local address 127.30.247.3:42953)
0925 05:00:37.699248 (+ 17248us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:37.699259 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:37.699295 (+ 36us) server_negotiation.cc:407] Connection header received
0925 05:00:37.699395 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:37.699404 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:37.699652 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:37.699902 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:00:37.700837 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.702188 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.704296 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.705628 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.705754 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:37.705989 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:37.707658 (+ 1669us) server_negotiation.cc:299] Negotiation successful
0925 05:00:37.707834 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17071,"thread_start_us":16906,"threads_started":1}
I20260925 05:00:37.708815 32752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:37.684148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:36603 (local address 127.0.0.1:45758)
0925 05:00:37.698881 (+ 14733us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:37.698902 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:37.699041 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:37.699802 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:37.699810 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:37.699839 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:00:37.700246 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:37.700264 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.702268 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.702275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:37.704718 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:37.704733 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.706755 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.706760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:37.706949 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:37.706981 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 05:00:37.707093 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 05:00:37.707244 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14577,"mutex_wait_us":80,"thread_start_us":13166,"threads_started":1}
I20260925 05:00:37.709249 32756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:37.684366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45758 (local address 127.30.247.2:36603)
0925 05:00:37.692571 (+ 8205us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:37.692585 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:37.699192 (+ 6607us) server_negotiation.cc:407] Connection header received
0925 05:00:37.699318 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:37.699326 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:37.699587 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:37.699786 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:00:37.700443 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.702128 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.704905 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.706294 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.706429 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:37.706615 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:37.707869 (+ 1254us) server_negotiation.cc:299] Negotiation successful
0925 05:00:37.708034 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7949,"thread_start_us":7820,"threads_started":1}
I20260925 05:00:37.711181 32407 tablet_service.cc:1467] Processing CreateTablet for tablet 142ffdf2c6e948fc95e6be877bbafd16 (DEFAULT_TABLE table=test-table [id=007be7b9df7144388f0cbec35b43274a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:00:37.711678 32674 tablet_service.cc:1467] Processing CreateTablet for tablet 142ffdf2c6e948fc95e6be877bbafd16 (DEFAULT_TABLE table=test-table [id=007be7b9df7144388f0cbec35b43274a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:00:37.712203 32541 tablet_service.cc:1467] Processing CreateTablet for tablet 142ffdf2c6e948fc95e6be877bbafd16 (DEFAULT_TABLE table=test-table [id=007be7b9df7144388f0cbec35b43274a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:00:37.712735 32407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 142ffdf2c6e948fc95e6be877bbafd16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:37.713024 32674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 142ffdf2c6e948fc95e6be877bbafd16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:37.713394 32541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 142ffdf2c6e948fc95e6be877bbafd16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:37.724292 32757 tablet_bootstrap.cc:492] T 142ffdf2c6e948fc95e6be877bbafd16 P 93448f97416746559e65ed42e9e65d64: Bootstrap starting.
I20260925 05:00:37.726799 32758 tablet_bootstrap.cc:492] T 142ffdf2c6e948fc95e6be877bbafd16 P b26c32c5e378412cad65fe4ac2abe998: Bootstrap starting.
I20260925 05:00:37.728257 32757 tablet_bootstrap.cc:654] T 142ffdf2c6e948fc95e6be877bbafd16 P 93448f97416746559e65ed42e9e65d64: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:37.729607 32757 log.cc:824] T 142ffdf2c6e948fc95e6be877bbafd16 P 93448f97416746559e65ed42e9e65d64: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:37.730258 32758 tablet_bootstrap.cc:654] T 142ffdf2c6e948fc95e6be877bbafd16 P b26c32c5e378412cad65fe4ac2abe998: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:37.730997 32759 tablet_bootstrap.cc:492] T 142ffdf2c6e948fc95e6be877bbafd16 P f88db44e0cab4f2b8a74c00624dac94f: Bootstrap starting.
I20260925 05:00:37.731464 32758 log.cc:824] T 142ffdf2c6e948fc95e6be877bbafd16 P b26c32c5e378412cad65fe4ac2abe998: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:37.733776 32758 tablet_bootstrap.cc:492] T 142ffdf2c6e948fc95e6be877bbafd16 P b26c32c5e378412cad65fe4ac2abe998: No bootstrap required, opened a new log
I20260925 05:00:37.733875 32757 tablet_bootstrap.cc:492] T 142ffdf2c6e948fc95e6be877bbafd16 P 93448f97416746559e65ed42e9e65d64: No bootstrap required, opened a new log
I20260925 05:00:37.734073 32758 ts_tablet_manager.cc:1397] T 142ffdf2c6e948fc95e6be877bbafd16 P b26c32c5e378412cad65fe4ac2abe998: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 05:00:37.734144 32757 ts_tablet_manager.cc:1397] T 142ffdf2c6e948fc95e6be877bbafd16 P 93448f97416746559e65ed42e9e65d64: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260925 05:00:37.735352 32759 tablet_bootstrap.cc:654] T 142ffdf2c6e948fc95e6be877bbafd16 P f88db44e0cab4f2b8a74c00624dac94f: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:37.736475 32759 log.cc:824] T 142ffdf2c6e948fc95e6be877bbafd16 P f88db44e0cab4f2b8a74c00624dac94f: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:37.739295 32759 tablet_bootstrap.cc:492] T 142ffdf2c6e948fc95e6be877bbafd16 P f88db44e0cab4f2b8a74c00624dac94f: No bootstrap required, opened a new log
I20260925 05:00:37.739581 32759 ts_tablet_manager.cc:1397] T 142ffdf2c6e948fc95e6be877bbafd16 P f88db44e0cab4f2b8a74c00624dac94f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260925 05:00:37.741493 32758 raft_consensus.cc:348] T 142ffdf2c6e948fc95e6be877bbafd16 P b26c32c5e378412cad65fe4ac2abe998 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88db44e0cab4f2b8a74c00624dac94f" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 42953 } } peers { permanent_uuid: "b26c32c5e378412cad65fe4ac2abe998" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46497 } } peers { permanent_uuid: "93448f97416746559e65ed42e9e65d64" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36603 } }
I20260925 05:00:37.741952 32758 raft_consensus.cc:374] T 142ffdf2c6e948fc95e6be877bbafd16 P b26c32c5e378412cad65fe4ac2abe998 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:37.742048 32758 raft_consensus.cc:729] T 142ffdf2c6e948fc95e6be877bbafd16 P b26c32c5e378412cad65fe4ac2abe998 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b26c32c5e378412cad65fe4ac2abe998, State: Initialized, Role: FOLLOWER
I20260925 05:00:37.742772 32758 consensus_queue.cc:260] T 142ffdf2c6e948fc95e6be877bbafd16 P b26c32c5e378412cad65fe4ac2abe998 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88db44e0cab4f2b8a74c00624dac94f" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 42953 } } peers { permanent_uuid: "b26c32c5e378412cad65fe4ac2abe998" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46497 } } peers { permanent_uuid: "93448f97416746559e65ed42e9e65d64" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36603 } }
I20260925 05:00:37.744441 32758 ts_tablet_manager.cc:1428] T 142ffdf2c6e948fc95e6be877bbafd16 P b26c32c5e378412cad65fe4ac2abe998: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260925 05:00:37.744225 32757 raft_consensus.cc:348] T 142ffdf2c6e948fc95e6be877bbafd16 P 93448f97416746559e65ed42e9e65d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88db44e0cab4f2b8a74c00624dac94f" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 42953 } } peers { permanent_uuid: "b26c32c5e378412cad65fe4ac2abe998" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46497 } } peers { permanent_uuid: "93448f97416746559e65ed42e9e65d64" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36603 } }
I20260925 05:00:37.744645 32757 raft_consensus.cc:374] T 142ffdf2c6e948fc95e6be877bbafd16 P 93448f97416746559e65ed42e9e65d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:37.744746 32757 raft_consensus.cc:729] T 142ffdf2c6e948fc95e6be877bbafd16 P 93448f97416746559e65ed42e9e65d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93448f97416746559e65ed42e9e65d64, State: Initialized, Role: FOLLOWER
I20260925 05:00:37.745311 32757 consensus_queue.cc:260] T 142ffdf2c6e948fc95e6be877bbafd16 P 93448f97416746559e65ed42e9e65d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88db44e0cab4f2b8a74c00624dac94f" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 42953 } } peers { permanent_uuid: "b26c32c5e378412cad65fe4ac2abe998" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46497 } } peers { permanent_uuid: "93448f97416746559e65ed42e9e65d64" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36603 } }
I20260925 05:00:37.746335 32606 heartbeater.cc:503] Master 127.30.247.62:35835 was elected leader, sending a full tablet report...
I20260925 05:00:37.748162 32757 ts_tablet_manager.cc:1428] T 142ffdf2c6e948fc95e6be877bbafd16 P 93448f97416746559e65ed42e9e65d64: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260925 05:00:37.751495 32759 raft_consensus.cc:348] T 142ffdf2c6e948fc95e6be877bbafd16 P f88db44e0cab4f2b8a74c00624dac94f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88db44e0cab4f2b8a74c00624dac94f" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 42953 } } peers { permanent_uuid: "b26c32c5e378412cad65fe4ac2abe998" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46497 } } peers { permanent_uuid: "93448f97416746559e65ed42e9e65d64" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36603 } }
I20260925 05:00:37.751879 32759 raft_consensus.cc:374] T 142ffdf2c6e948fc95e6be877bbafd16 P f88db44e0cab4f2b8a74c00624dac94f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:37.751952 32759 raft_consensus.cc:729] T 142ffdf2c6e948fc95e6be877bbafd16 P f88db44e0cab4f2b8a74c00624dac94f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f88db44e0cab4f2b8a74c00624dac94f, State: Initialized, Role: FOLLOWER
I20260925 05:00:37.752379 32759 consensus_queue.cc:260] T 142ffdf2c6e948fc95e6be877bbafd16 P f88db44e0cab4f2b8a74c00624dac94f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88db44e0cab4f2b8a74c00624dac94f" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 42953 } } peers { permanent_uuid: "b26c32c5e378412cad65fe4ac2abe998" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46497 } } peers { permanent_uuid: "93448f97416746559e65ed42e9e65d64" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36603 } }
I20260925 05:00:37.753330 32739 heartbeater.cc:503] Master 127.30.247.62:35835 was elected leader, sending a full tablet report...
I20260925 05:00:37.753681 32759 ts_tablet_manager.cc:1428] T 142ffdf2c6e948fc95e6be877bbafd16 P f88db44e0cab4f2b8a74c00624dac94f: Time spent starting tablet: real 0.014s user 0.005s sys 0.008s
W20260925 05:00:37.854385 32473 tablet.cc:2369] T 142ffdf2c6e948fc95e6be877bbafd16 P b26c32c5e378412cad65fe4ac2abe998: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:00:37.855744 32740 tablet.cc:2369] T 142ffdf2c6e948fc95e6be877bbafd16 P f88db44e0cab4f2b8a74c00624dac94f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:00:37.859364 32607 tablet.cc:2369] T 142ffdf2c6e948fc95e6be877bbafd16 P 93448f97416746559e65ed42e9e65d64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:00:37.925015 32764 raft_consensus.cc:482] T 142ffdf2c6e948fc95e6be877bbafd16 P 93448f97416746559e65ed42e9e65d64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:00:37.925354 32764 raft_consensus.cc:504] T 142ffdf2c6e948fc95e6be877bbafd16 P 93448f97416746559e65ed42e9e65d64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88db44e0cab4f2b8a74c00624dac94f" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 42953 } } peers { permanent_uuid: "b26c32c5e378412cad65fe4ac2abe998" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46497 } } peers { permanent_uuid: "93448f97416746559e65ed42e9e65d64" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36603 } }
I20260925 05:00:37.927101 32764 leader_election.cc:288] T 142ffdf2c6e948fc95e6be877bbafd16 P 93448f97416746559e65ed42e9e65d64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f88db44e0cab4f2b8a74c00624dac94f (127.30.247.3:42953), b26c32c5e378412cad65fe4ac2abe998 (127.30.247.1:46497)
I20260925 05:00:37.936717 32754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:37.927688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:60211 (local address 127.30.247.1:46497)
0925 05:00:37.927895 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:37.927901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:37.928626 (+ 725us) server_negotiation.cc:407] Connection header received
0925 05:00:37.928949 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:37.928957 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:37.929041 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:37.929107 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:00:37.930145 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.931248 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.934308 (+ 3060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.935601 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.935748 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:37.935822 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:37.936316 (+ 494us) server_negotiation.cc:299] Negotiation successful
0925 05:00:37.936449 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 05:00:37.936731 32753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:37.927546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:34661 (local address 127.30.247.3:42953)
0925 05:00:37.927746 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:37.927755 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:37.929046 (+ 1291us) server_negotiation.cc:407] Connection header received
0925 05:00:37.929197 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:37.929201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:37.929259 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:37.929332 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:00:37.930132 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.931248 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.934308 (+ 3060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.935579 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.935748 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:37.935822 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:37.936316 (+ 494us) server_negotiation.cc:299] Negotiation successful
0925 05:00:37.936446 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 05:00:37.936936 32767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:37.928140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:42953 (local address 127.30.247.2:34661)
0925 05:00:37.928943 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:37.928959 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:37.929077 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:37.929364 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:37.929368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:37.929391 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:00:37.929885 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:37.929902 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.931439 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.931445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:37.934111 (+ 2666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:37.934127 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.935844 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.935857 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:37.936086 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:37.936147 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 05:00:37.936240 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 05:00:37.936410 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":207,"thread_start_us":483,"threads_started":1}
I20260925 05:00:37.937120 32766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:37.927539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:46497 (local address 127.30.247.2:60211)
0925 05:00:37.928457 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:37.928485 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:37.928765 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:37.929299 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:37.929326 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:37.929354 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:00:37.929953 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:37.929964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.931427 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.931434 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:37.934119 (+ 2685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:37.934127 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.935867 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.935872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:37.936092 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:37.936142 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 05:00:37.936240 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 05:00:37.936411 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":242,"thread_start_us":545,"threads_started":1}
I20260925 05:00:37.937973 32694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142ffdf2c6e948fc95e6be877bbafd16" candidate_uuid: "93448f97416746559e65ed42e9e65d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f88db44e0cab4f2b8a74c00624dac94f" is_pre_election: true
I20260925 05:00:37.938081 32427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142ffdf2c6e948fc95e6be877bbafd16" candidate_uuid: "93448f97416746559e65ed42e9e65d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b26c32c5e378412cad65fe4ac2abe998" is_pre_election: true
I20260925 05:00:37.938476 32694 raft_consensus.cc:2454] T 142ffdf2c6e948fc95e6be877bbafd16 P f88db44e0cab4f2b8a74c00624dac94f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93448f97416746559e65ed42e9e65d64 in term 0.
I20260925 05:00:37.938576 32427 raft_consensus.cc:2454] T 142ffdf2c6e948fc95e6be877bbafd16 P b26c32c5e378412cad65fe4ac2abe998 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93448f97416746559e65ed42e9e65d64 in term 0.
I20260925 05:00:37.939200 32495 leader_election.cc:302] T 142ffdf2c6e948fc95e6be877bbafd16 P 93448f97416746559e65ed42e9e65d64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93448f97416746559e65ed42e9e65d64, f88db44e0cab4f2b8a74c00624dac94f; no voters:
I20260925 05:00:37.939765 32764 raft_consensus.cc:2781] T 142ffdf2c6e948fc95e6be877bbafd16 P 93448f97416746559e65ed42e9e65d64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:00:37.939893 32764 raft_consensus.cc:482] T 142ffdf2c6e948fc95e6be877bbafd16 P 93448f97416746559e65ed42e9e65d64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:00:37.940012 32764 raft_consensus.cc:3037] T 142ffdf2c6e948fc95e6be877bbafd16 P 93448f97416746559e65ed42e9e65d64 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:37.942183 32764 raft_consensus.cc:504] T 142ffdf2c6e948fc95e6be877bbafd16 P 93448f97416746559e65ed42e9e65d64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88db44e0cab4f2b8a74c00624dac94f" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 42953 } } peers { permanent_uuid: "b26c32c5e378412cad65fe4ac2abe998" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46497 } } peers { permanent_uuid: "93448f97416746559e65ed42e9e65d64" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36603 } }
I20260925 05:00:37.942808 32764 leader_election.cc:288] T 142ffdf2c6e948fc95e6be877bbafd16 P 93448f97416746559e65ed42e9e65d64 [CANDIDATE]: Term 1 election: Requested vote from peers f88db44e0cab4f2b8a74c00624dac94f (127.30.247.3:42953), b26c32c5e378412cad65fe4ac2abe998 (127.30.247.1:46497)
I20260925 05:00:37.943244 32694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142ffdf2c6e948fc95e6be877bbafd16" candidate_uuid: "93448f97416746559e65ed42e9e65d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f88db44e0cab4f2b8a74c00624dac94f"
I20260925 05:00:37.943245 32427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142ffdf2c6e948fc95e6be877bbafd16" candidate_uuid: "93448f97416746559e65ed42e9e65d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b26c32c5e378412cad65fe4ac2abe998"
I20260925 05:00:37.943432 32694 raft_consensus.cc:3037] T 142ffdf2c6e948fc95e6be877bbafd16 P f88db44e0cab4f2b8a74c00624dac94f [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:37.943449 32427 raft_consensus.cc:3037] T 142ffdf2c6e948fc95e6be877bbafd16 P b26c32c5e378412cad65fe4ac2abe998 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:37.945725 32427 raft_consensus.cc:2454] T 142ffdf2c6e948fc95e6be877bbafd16 P b26c32c5e378412cad65fe4ac2abe998 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93448f97416746559e65ed42e9e65d64 in term 1.
I20260925 05:00:37.945729 32694 raft_consensus.cc:2454] T 142ffdf2c6e948fc95e6be877bbafd16 P f88db44e0cab4f2b8a74c00624dac94f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93448f97416746559e65ed42e9e65d64 in term 1.
I20260925 05:00:37.946120 32497 leader_election.cc:302] T 142ffdf2c6e948fc95e6be877bbafd16 P 93448f97416746559e65ed42e9e65d64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93448f97416746559e65ed42e9e65d64, b26c32c5e378412cad65fe4ac2abe998; no voters:
I20260925 05:00:37.946393 32764 raft_consensus.cc:2781] T 142ffdf2c6e948fc95e6be877bbafd16 P 93448f97416746559e65ed42e9e65d64 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:00:37.946839 32764 raft_consensus.cc:686] T 142ffdf2c6e948fc95e6be877bbafd16 P 93448f97416746559e65ed42e9e65d64 [term 1 LEADER]: Becoming Leader. State: Replica: 93448f97416746559e65ed42e9e65d64, State: Running, Role: LEADER
I20260925 05:00:37.947345 32764 consensus_queue.cc:237] T 142ffdf2c6e948fc95e6be877bbafd16 P 93448f97416746559e65ed42e9e65d64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88db44e0cab4f2b8a74c00624dac94f" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 42953 } } peers { permanent_uuid: "b26c32c5e378412cad65fe4ac2abe998" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46497 } } peers { permanent_uuid: "93448f97416746559e65ed42e9e65d64" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36603 } }
I20260925 05:00:37.951721 32283 catalog_manager.cc:5495] T 142ffdf2c6e948fc95e6be877bbafd16 P 93448f97416746559e65ed42e9e65d64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93448f97416746559e65ed42e9e65d64 (127.30.247.2). New cstate: current_term: 1 leader_uuid: "93448f97416746559e65ed42e9e65d64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88db44e0cab4f2b8a74c00624dac94f" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 42953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b26c32c5e378412cad65fe4ac2abe998" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93448f97416746559e65ed42e9e65d64" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36603 } health_report { overall_health: HEALTHY } } }
Sep 25 05:00:37 dist-test-slave-5n86 krb5kdc[32238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.1@KRBTEST.COM
I20260925 05:00:38.002974 32754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:37.988199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44514 (local address 127.30.247.1:46497)
0925 05:00:37.988450 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:37.988458 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:37.988879 (+ 421us) server_negotiation.cc:407] Connection header received
0925 05:00:37.989086 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:37.989097 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:37.989161 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:37.989266 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:37.990708 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.991443 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.992705 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:37.993132 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:37.993261 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:37.997219 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:37.997233 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:37.997251 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:37.997284 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:38.000294 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:38.001075 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:38.001083 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:38.001085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:38.001228 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:38.001536 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:38.001539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:38.001541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:38.001909 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:38.002129 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:38.002418 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 05:00:38.002579 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Sep 25 05:00:38 dist-test-slave-5n86 krb5kdc[32238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312438, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 05:00:38 dist-test-slave-5n86 krb5kdc[32238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312438, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.247.1@KRBTEST.COM
I20260925 05:00:38.052831 32754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:38.038069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44518 (local address 127.30.247.1:46497)
0925 05:00:38.038264 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:38.038270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:38.038526 (+ 256us) server_negotiation.cc:407] Connection header received
0925 05:00:38.038681 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:38.038685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:38.038750 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:38.038847 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:38.040453 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:38.041266 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:38.043202 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:38.043750 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:38.043847 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:38.047526 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:38.047531 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:38.047534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:38.047565 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:38.050418 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:38.050957 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:38.050961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:38.050963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:38.051100 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:38.051618 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:38.051622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:38.051625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:38.051892 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:38.052083 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:38.052306 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 05:00:38.052464 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
Sep 25 05:00:38 dist-test-slave-5n86 krb5kdc[32238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 05:00:38 dist-test-slave-5n86 krb5kdc[32238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.1@KRBTEST.COM
I20260925 05:00:38.095104 32754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:38.081435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44522 (local address 127.30.247.1:46497)
0925 05:00:38.081647 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:38.081656 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:38.081964 (+ 308us) server_negotiation.cc:407] Connection header received
0925 05:00:38.082181 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:38.082187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:38.082251 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:38.082361 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:38.083685 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:38.084358 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:38.085626 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:38.086058 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:38.086192 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:38.089970 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:38.089975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:38.089978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:38.090017 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:38.092528 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:38.093187 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:38.093195 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:38.093201 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:38.093326 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:38.093673 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:38.093687 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:38.093690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:38.093989 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:38.094245 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:38.094532 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 05:00:38.094783 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 05:00:38.096859 32387 tablet_service.cc:2908] Rejecting scan request for tablet 142ffdf2c6e948fc95e6be877bbafd16: Uninitialized: clean time has not yet been initialized
I20260925 05:00:38.099815 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 32344
I20260925 05:00:38.113152 32754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:38.100743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44530 (local address 127.30.247.1:46497)
0925 05:00:38.100947 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:38.100957 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:38.101259 (+ 302us) server_negotiation.cc:407] Connection header received
0925 05:00:38.101367 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:38.101370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:38.101436 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:38.101512 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:38.102581 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:38.103232 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:38.104387 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:38.104864 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:38.104961 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:38.107369 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:38.107376 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:38.107380 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:38.107417 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:38.110563 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:38.111171 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:38.111176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:38.111180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:38.111324 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:38.111676 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:38.111680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:38.111685 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:38.112064 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:38.112340 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:38.112611 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 05:00:38.112830 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 05:00:38.113826 32467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:00:38.204661 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 32344
I20260925 05:00:38.221194 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 32478
Sep 25 05:00:38 dist-test-slave-5n86 krb5kdc[32238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.2@KRBTEST.COM
I20260925 05:00:38.235935 314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:38.222053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45774 (local address 127.30.247.2:36603)
0925 05:00:38.222470 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:38.222475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:38.222490 (+ 15us) server_negotiation.cc:407] Connection header received
0925 05:00:38.222543 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:38.222547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:38.222627 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:38.222723 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:38.223833 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:38.224498 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:38.225867 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:38.226292 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:38.226424 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:38.230314 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:38.230326 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:38.230340 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:38.230377 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:38.233344 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:38.233984 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:38.233991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:38.233993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:38.234113 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:38.234461 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:38.234467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:38.234471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:38.234790 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:38.235083 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:38.235368 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 05:00:38.235600 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":195,"threads_started":1}
I20260925 05:00:38.236665 32601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:00:38.325263 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 32478
I20260925 05:00:38.342392 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 32611
Sep 25 05:00:38 dist-test-slave-5n86 krb5kdc[32238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.3@KRBTEST.COM
I20260925 05:00:38.356673 32753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:38.343142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57362 (local address 127.30.247.3:42953)
0925 05:00:38.343308 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:38.343313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:38.343330 (+ 17us) server_negotiation.cc:407] Connection header received
0925 05:00:38.343492 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:38.343496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:38.343563 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:38.343647 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:38.344865 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:38.345581 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:38.346815 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:38.347182 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:38.347279 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:38.351247 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:38.351261 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:38.351277 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:38.351310 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:38.354036 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:38.354626 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:38.354634 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:38.354636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:38.354772 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:38.355127 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:38.355131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:38.355134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:38.355510 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:38.355739 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:38.356059 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 05:00:38.356229 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 05:00:38.357319 32734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:00:38.442799 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 32611
I20260925 05:00:38.458595 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 32251
I20260925 05:00:38.459749 32313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:00:38.566298 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 32251
2026-09-25T05:00:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4196 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:00:38 dist-test-slave-5n86 krb5kdc[320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:00:38 dist-test-slave-5n86 krb5kdc[320](info): set up 2 sockets
Sep 25 05:00:38 dist-test-slave-5n86 krb5kdc[320](info): commencing operation
krb5kdc: starting...
W20260925 05:00:39.644735 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.005s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:00:39 dist-test-slave-5n86 krb5kdc[320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:00:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:00:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:00:39.815745 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:45447
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:45447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:40.125749 333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:40.126027 333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:40.126195 333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:40.133188 333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:00:40.133278 333 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 05:00:40.133366 333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:40.133453 333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:40.133507 333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:00:40.133631 333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:00:40.143088 333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:45447
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:45447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:40.144299 333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:40.145645 333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:40.156935 338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:40.156941 341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:40.157145 339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:40.158069 333 server_base.cc:1034] running on GCE node
I20260925 05:00:40.158820 333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:40.160135 333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:40.161372 333 hybrid_clock.cc:648] HybridClock initialized: now 1790312440161305 us; error 45 us; skew 500 ppm
I20260925 05:00:40.161720 333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:40 dist-test-slave-5n86 krb5kdc[320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312440, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:40.164690 333 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:00:40.166065 333 webserver.cc:460] Webserver started at http://127.30.247.62:33411/ using document root <none> and password file <none>
I20260925 05:00:40.166740 333 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:40.166870 333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:40.167176 333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:40.170109 333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "e065cd42faf44fe08e2635e69533842a"
format_stamp: "Formatted at 2026-09-25 05:00:40 on dist-test-slave-5n86"
server_key: "aa3a12df376bc02dd4b87631baf6a28c"
server_key_iv: "18042637348ab38865e83df32391572b"
server_key_version: "encryptionkey@0"
I20260925 05:00:40.170843 333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "e065cd42faf44fe08e2635e69533842a"
format_stamp: "Formatted at 2026-09-25 05:00:40 on dist-test-slave-5n86"
server_key: "aa3a12df376bc02dd4b87631baf6a28c"
server_key_iv: "18042637348ab38865e83df32391572b"
server_key_version: "encryptionkey@0"
I20260925 05:00:40.175608 333 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260925 05:00:40.178817 348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:40.180075 333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:00:40.180253 333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "e065cd42faf44fe08e2635e69533842a"
format_stamp: "Formatted at 2026-09-25 05:00:40 on dist-test-slave-5n86"
server_key: "aa3a12df376bc02dd4b87631baf6a28c"
server_key_iv: "18042637348ab38865e83df32391572b"
server_key_version: "encryptionkey@0"
I20260925 05:00:40.180456 333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:40.195348 333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:40.199059 333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:40.199313 333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:40.220256 333 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:45447
I20260925 05:00:40.220279 409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:45447 every 8 connection(s)
I20260925 05:00:40.221534 333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:00:40.225044 410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:40.230254 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 333
I20260925 05:00:40.230479 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:00:40.231120 31708 external_mini_cluster.cc:1428] Setting key 801038f51d41ea07fe925c1b90dc88a6
I20260925 05:00:40.234959 410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e065cd42faf44fe08e2635e69533842a: Bootstrap starting.
I20260925 05:00:40.238466 410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e065cd42faf44fe08e2635e69533842a: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:40.239528 410 log.cc:824] T 00000000000000000000000000000000 P e065cd42faf44fe08e2635e69533842a: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:40.242290 410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e065cd42faf44fe08e2635e69533842a: No bootstrap required, opened a new log
Sep 25 05:00:40 dist-test-slave-5n86 krb5kdc[320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:00:40.250550 410 raft_consensus.cc:348] T 00000000000000000000000000000000 P e065cd42faf44fe08e2635e69533842a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e065cd42faf44fe08e2635e69533842a" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45447 } }
I20260925 05:00:40.250921 410 raft_consensus.cc:374] T 00000000000000000000000000000000 P e065cd42faf44fe08e2635e69533842a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:40.251060 410 raft_consensus.cc:729] T 00000000000000000000000000000000 P e065cd42faf44fe08e2635e69533842a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e065cd42faf44fe08e2635e69533842a, State: Initialized, Role: FOLLOWER
I20260925 05:00:40.251734 410 consensus_queue.cc:260] T 00000000000000000000000000000000 P e065cd42faf44fe08e2635e69533842a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e065cd42faf44fe08e2635e69533842a" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45447 } }
I20260925 05:00:40.252002 410 raft_consensus.cc:388] T 00000000000000000000000000000000 P e065cd42faf44fe08e2635e69533842a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:00:40.252136 410 raft_consensus.cc:482] T 00000000000000000000000000000000 P e065cd42faf44fe08e2635e69533842a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:00:40.252323 410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e065cd42faf44fe08e2635e69533842a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:40.253319 413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:40.232945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45040 (local address 127.30.247.62:45447)
0925 05:00:40.233706 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:40.233717 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:40.233756 (+ 39us) server_negotiation.cc:407] Connection header received
0925 05:00:40.235379 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:40.235403 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:40.235769 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:40.236058 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:40.237766 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:40.239076 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:40.240670 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:40.241291 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:40.241411 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:40.246005 (+ 4594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:40.246012 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:40.246026 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:40.246061 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:40.249110 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:40.249753 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:40.249759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:40.249766 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:40.249930 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:40.250305 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:40.250309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:40.250311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:40.250803 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:40.251129 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:40.251488 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 05:00:40.251895 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":392,"threads_started":1}
I20260925 05:00:40.255386 410 raft_consensus.cc:504] T 00000000000000000000000000000000 P e065cd42faf44fe08e2635e69533842a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e065cd42faf44fe08e2635e69533842a" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45447 } }
I20260925 05:00:40.255899 410 leader_election.cc:302] T 00000000000000000000000000000000 P e065cd42faf44fe08e2635e69533842a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e065cd42faf44fe08e2635e69533842a; no voters:
I20260925 05:00:40.256472 410 leader_election.cc:288] T 00000000000000000000000000000000 P e065cd42faf44fe08e2635e69533842a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:00:40.256682 415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e065cd42faf44fe08e2635e69533842a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:00:40.257370 415 raft_consensus.cc:686] T 00000000000000000000000000000000 P e065cd42faf44fe08e2635e69533842a [term 1 LEADER]: Becoming Leader. State: Replica: e065cd42faf44fe08e2635e69533842a, State: Running, Role: LEADER
I20260925 05:00:40.257871 415 consensus_queue.cc:237] T 00000000000000000000000000000000 P e065cd42faf44fe08e2635e69533842a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e065cd42faf44fe08e2635e69533842a" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45447 } }
I20260925 05:00:40.258369 410 sys_catalog.cc:564] T 00000000000000000000000000000000 P e065cd42faf44fe08e2635e69533842a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:00:40.261030 417 sys_catalog.cc:455] T 00000000000000000000000000000000 P e065cd42faf44fe08e2635e69533842a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e065cd42faf44fe08e2635e69533842a. Latest consensus state: current_term: 1 leader_uuid: "e065cd42faf44fe08e2635e69533842a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e065cd42faf44fe08e2635e69533842a" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45447 } } }
I20260925 05:00:40.261009 416 sys_catalog.cc:455] T 00000000000000000000000000000000 P e065cd42faf44fe08e2635e69533842a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e065cd42faf44fe08e2635e69533842a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e065cd42faf44fe08e2635e69533842a" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45447 } } }
I20260925 05:00:40.261422 417 sys_catalog.cc:458] T 00000000000000000000000000000000 P e065cd42faf44fe08e2635e69533842a [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:40.261430 416 sys_catalog.cc:458] T 00000000000000000000000000000000 P e065cd42faf44fe08e2635e69533842a [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:40.265916 421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:00:40.270972 421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:00:40.280045 421 catalog_manager.cc:1347] Generated new cluster ID: 7684126a48344d57a38ba9686ef4fec9
I20260925 05:00:40.280151 421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:00:40.286861 421 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:00:40.288076 421 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:00:40.294727 421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e065cd42faf44fe08e2635e69533842a: Generated new TSK 0
I20260925 05:00:40.295408 421 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
I20260925 05:00:40.345036 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:0
--local_ip_for_outbound_sockets=127.30.247.1
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:45447
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:40.658355 436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:40.658697 436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:40.658860 436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:40.666201 436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:40.666345 436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:40.666491 436 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:00:40.666638 436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:00:40.676445 436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.247.62:45447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:40.677688 436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:40.678972 436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:40.691934 444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:40.692011 441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:40.691983 442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:40.692811 436 server_base.cc:1034] running on GCE node
I20260925 05:00:40.693389 436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:40.694198 436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:40.695426 436 hybrid_clock.cc:648] HybridClock initialized: now 1790312440695347 us; error 41 us; skew 500 ppm
I20260925 05:00:40.695709 436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:40 dist-test-slave-5n86 krb5kdc[320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312440, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:40.698705 436 init.cc:377] Logged in from keytab as kudu/127.30.247.1@KRBTEST.COM (short username kudu)
I20260925 05:00:40.700100 436 webserver.cc:460] Webserver started at http://127.30.247.1:33061/ using document root <none> and password file <none>
I20260925 05:00:40.700834 436 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:40.700956 436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:40.701259 436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:40.704264 436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-0/data/instance:
uuid: "9b3474e03feb4e0f96eb9dc84b4f568c"
format_stamp: "Formatted at 2026-09-25 05:00:40 on dist-test-slave-5n86"
server_key: "c166b212954826b445ecc7aeeead0e4c"
server_key_iv: "74df1708d393deba9070a7b304165ba5"
server_key_version: "encryptionkey@0"
I20260925 05:00:40.705075 436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance:
uuid: "9b3474e03feb4e0f96eb9dc84b4f568c"
format_stamp: "Formatted at 2026-09-25 05:00:40 on dist-test-slave-5n86"
server_key: "c166b212954826b445ecc7aeeead0e4c"
server_key_iv: "74df1708d393deba9070a7b304165ba5"
server_key_version: "encryptionkey@0"
I20260925 05:00:40.709812 436 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 05:00:40.712968 451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:40.714110 436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 05:00:40.714272 436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "9b3474e03feb4e0f96eb9dc84b4f568c"
format_stamp: "Formatted at 2026-09-25 05:00:40 on dist-test-slave-5n86"
server_key: "c166b212954826b445ecc7aeeead0e4c"
server_key_iv: "74df1708d393deba9070a7b304165ba5"
server_key_version: "encryptionkey@0"
I20260925 05:00:40.714479 436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:40.730764 436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:40.734462 436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:40.734724 436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:40.737239 436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:00:40.737346 436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:40.737524 436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:00:40.737622 436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 05:00:40 dist-test-slave-5n86 krb5kdc[320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312440, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:00:40.759948 413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:40.737750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:39967 (local address 127.30.247.62:45447)
0925 05:00:40.737947 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:40.737953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:40.739316 (+ 1363us) server_negotiation.cc:407] Connection header received
0925 05:00:40.741271 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:40.741279 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:40.741386 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:40.741508 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:40.743614 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:40.744653 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:40.747147 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:40.747737 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:40.747869 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:40.753355 (+ 5486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:40.753360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:40.753363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:40.753394 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:40.756707 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:40.757413 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:40.757419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:40.757423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:40.757550 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:40.757985 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:40.757991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:40.757995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:40.758332 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:40.758480 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:40.759332 (+ 852us) server_negotiation.cc:299] Negotiation successful
0925 05:00:40.759560 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 05:00:40.761490 460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:40.738258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:45447 (local address 127.30.247.1:39967)
0925 05:00:40.739114 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:40.739167 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:40.741012 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:40.741757 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:40.741767 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:40.742610 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:40.743445 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:40.743464 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:40.744849 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:40.744857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:40.746988 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:40.747003 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:40.748028 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:40.748039 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:40.748381 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:40.749302 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:40.749332 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:40.753053 (+ 3721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:40.756844 (+ 3791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:40.756852 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:40.756869 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:40.757289 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:40.757659 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:40.757667 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:40.757671 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:40.757873 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:40.758499 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:40.758508 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:40.758942 (+ 434us) client_negotiation.cc:769] Sending connection context
0925 05:00:40.759304 (+ 362us) client_negotiation.cc:240] Negotiation successful
0925 05:00:40.759846 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":365,"threads_started":1}
I20260925 05:00:40.780906 413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:40.767280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:44465 (local address 127.30.247.62:45447)
0925 05:00:40.767490 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:40.767499 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:40.768104 (+ 605us) server_negotiation.cc:407] Connection header received
0925 05:00:40.768302 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:40.768306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:40.768362 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:40.768438 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:40.769950 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:40.770875 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:40.772973 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:40.773379 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:40.773521 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:40.775277 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:40.775282 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:40.775285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:40.775314 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:40.778020 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:40.779002 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:40.779010 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:40.779016 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:40.779139 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:40.779479 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:40.779486 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:40.779489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:40.779750 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:40.779909 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:40.780279 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 05:00:40.780488 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 05:00:40.781030 523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:40.767237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:45447 (local address 127.30.247.1:44465)
0925 05:00:40.767977 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:40.768009 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:40.768185 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:40.768519 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:40.768524 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:40.769263 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:40.769767 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:40.769784 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:40.771083 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:40.771091 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:40.772828 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:40.772843 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:40.773559 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:40.773566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:40.773719 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:40.774398 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:40.774414 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:40.775085 (+ 671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:40.778200 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:40.778206 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:40.778209 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:40.778873 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:40.779246 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:40.779252 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:40.779256 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:40.779378 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:40.779866 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:40.779871 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:40.780045 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 05:00:40.780192 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 05:00:40.780494 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":384,"threads_started":1}
I20260925 05:00:40.790773 436 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:33069
I20260925 05:00:40.790795 570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:33069 every 8 connection(s)
I20260925 05:00:40.791936 436 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:00:40.793263 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 436
I20260925 05:00:40.793437 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance
I20260925 05:00:40.793985 31708 external_mini_cluster.cc:1428] Setting key eb4c9838bf620c9e6fc6ed84c4872466
I20260925 05:00:40.795715 571 heartbeater.cc:347] Connected to a master server at 127.30.247.62:45447
I20260925 05:00:40.795958 571 heartbeater.cc:464] Registering TS with master...
I20260925 05:00:40.796617 571 heartbeater.cc:511] Master 127.30.247.62:45447 requested a full tablet report, sending...
I20260925 05:00:40.798646 365 ts_manager.cc:194] Registered new tserver with Master: 9b3474e03feb4e0f96eb9dc84b4f568c (127.30.247.1:33069)
I20260925 05:00:40.800793 365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.1@KRBTEST.COM'} at 127.30.247.1:39967
WARNING: no policy specified for kudu/127.30.247.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
I20260925 05:00:40.832060 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:0
--local_ip_for_outbound_sockets=127.30.247.2
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:45447
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:41.137801 575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:41.138057 575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:41.138157 575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:41.145066 575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:41.145164 575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:41.145242 575 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:00:41.145323 575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:00:41.154381 575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.247.62:45447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:41.155500 575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:41.156788 575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:41.169613 580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:41.169694 583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:41.169723 581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:41.170560 575 server_base.cc:1034] running on GCE node
I20260925 05:00:41.171101 575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:41.171880 575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:41.173118 575 hybrid_clock.cc:648] HybridClock initialized: now 1790312441173047 us; error 27 us; skew 500 ppm
I20260925 05:00:41.173416 575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:41 dist-test-slave-5n86 krb5kdc[320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312441, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:41.179447 575 init.cc:377] Logged in from keytab as kudu/127.30.247.2@KRBTEST.COM (short username kudu)
I20260925 05:00:41.180830 575 webserver.cc:460] Webserver started at http://127.30.247.2:43055/ using document root <none> and password file <none>
I20260925 05:00:41.181475 575 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:41.181604 575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:41.181913 575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:41.184764 575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-1/data/instance:
uuid: "7a2800457a6e43d9a8d54a9240e98c1e"
format_stamp: "Formatted at 2026-09-25 05:00:41 on dist-test-slave-5n86"
server_key: "c88d72825c7b9c1ba9e37ca8be164a03"
server_key_iv: "8b74f05dcb30ebdd445c835e6a1fb228"
server_key_version: "encryptionkey@0"
I20260925 05:00:41.185520 575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance:
uuid: "7a2800457a6e43d9a8d54a9240e98c1e"
format_stamp: "Formatted at 2026-09-25 05:00:41 on dist-test-slave-5n86"
server_key: "c88d72825c7b9c1ba9e37ca8be164a03"
server_key_iv: "8b74f05dcb30ebdd445c835e6a1fb228"
server_key_version: "encryptionkey@0"
I20260925 05:00:41.190313 575 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 05:00:41.193743 590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:41.195004 575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 05:00:41.195184 575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "7a2800457a6e43d9a8d54a9240e98c1e"
format_stamp: "Formatted at 2026-09-25 05:00:41 on dist-test-slave-5n86"
server_key: "c88d72825c7b9c1ba9e37ca8be164a03"
server_key_iv: "8b74f05dcb30ebdd445c835e6a1fb228"
server_key_version: "encryptionkey@0"
I20260925 05:00:41.195402 575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:41.213299 575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:41.217144 575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:41.217396 575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:41.219962 575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:00:41.220057 575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:41.220197 575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:00:41.220275 575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 05:00:41 dist-test-slave-5n86 krb5kdc[320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312441, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:00:41.251046 413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:41.224706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:48933 (local address 127.30.247.62:45447)
0925 05:00:41.224936 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:41.224945 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:41.229662 (+ 4717us) server_negotiation.cc:407] Connection header received
0925 05:00:41.232626 (+ 2964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:41.232631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:41.232724 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:41.232834 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:41.234305 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.235644 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.238349 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.239155 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.239282 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:41.243495 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:41.243503 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:41.243511 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:41.243554 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:41.246887 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:41.247950 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:41.247955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:41.247958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:41.248090 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:41.248700 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:41.248708 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:41.248713 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:41.249130 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:41.249294 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:41.250470 (+ 1176us) server_negotiation.cc:299] Negotiation successful
0925 05:00:41.250717 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 05:00:41.252213 599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:41.221186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:45447 (local address 127.30.247.2:48933)
0925 05:00:41.229513 (+ 8327us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:41.229561 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:41.232346 (+ 2785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:41.233171 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:41.233179 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:41.233643 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:41.234145 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:41.234161 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.235877 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.235882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:41.238212 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:41.238224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.239381 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.239388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:41.239575 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:41.240210 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:41.240231 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:41.243181 (+ 2950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:41.247050 (+ 3869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:41.247062 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:41.247083 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:41.247788 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:41.248202 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:41.248210 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:41.248216 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:41.248502 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:41.249374 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:41.249383 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:41.249888 (+ 505us) client_negotiation.cc:769] Sending connection context
0925 05:00:41.250262 (+ 374us) client_negotiation.cc:240] Negotiation successful
0925 05:00:41.250738 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8090,"thread_start_us":7927,"threads_started":1}
I20260925 05:00:41.273690 665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:41.257981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:45447 (local address 127.30.247.2:59981)
0925 05:00:41.258802 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:41.258826 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:41.258962 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:41.259484 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:41.259492 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:41.260120 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:41.260440 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:41.260449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.262255 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.262262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:41.263594 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:41.263609 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.264787 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.264791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:41.264896 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:41.265504 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:41.265520 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:41.266261 (+ 741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:41.270993 (+ 4732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:41.270997 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:41.271000 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:41.271449 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:41.271943 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:41.271949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:41.271956 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:41.272090 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:41.272862 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:41.272865 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:41.273008 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 05:00:41.273129 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 05:00:41.273360 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":524,"threads_started":1}
I20260925 05:00:41.274097 413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:41.257869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:59981 (local address 127.30.247.62:45447)
0925 05:00:41.258068 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:41.258075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:41.258899 (+ 824us) server_negotiation.cc:407] Connection header received
0925 05:00:41.259081 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:41.259086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:41.259155 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:41.259293 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:41.260613 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.262068 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.263736 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.264604 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.264765 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:41.266478 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:41.266488 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:41.266495 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:41.266540 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:41.270836 (+ 4296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:41.271576 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:41.271585 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:41.271591 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:41.271787 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:41.272199 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:41.272207 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:41.272212 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:41.272721 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:41.272893 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:41.273301 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 05:00:41.273604 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 05:00:41.286377 575 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:34935
I20260925 05:00:41.286414 709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:34935 every 8 connection(s)
I20260925 05:00:41.287595 575 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:00:41.289641 710 heartbeater.cc:347] Connected to a master server at 127.30.247.62:45447
I20260925 05:00:41.289878 710 heartbeater.cc:464] Registering TS with master...
I20260925 05:00:41.290490 710 heartbeater.cc:511] Master 127.30.247.62:45447 requested a full tablet report, sending...
I20260925 05:00:41.291792 365 ts_manager.cc:194] Registered new tserver with Master: 7a2800457a6e43d9a8d54a9240e98c1e (127.30.247.2:34935)
I20260925 05:00:41.292702 365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.2@KRBTEST.COM'} at 127.30.247.2:48933
I20260925 05:00:41.297183 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 575
I20260925 05:00:41.297338 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance
I20260925 05:00:41.297713 31708 external_mini_cluster.cc:1428] Setting key e2a758a87651b63183c95682943c6029
WARNING: no policy specified for kudu/127.30.247.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
I20260925 05:00:41.335475 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:0
--local_ip_for_outbound_sockets=127.30.247.3
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:45447
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:41.657604 714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:41.657864 714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:41.657958 714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:41.665347 714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:41.665445 714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:41.665525 714 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:00:41.665602 714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:00:41.674803 714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.247.62:45447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:41.675927 714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:41.677157 714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:41.689683 719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:41.689532 722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:41.689607 720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:41.691056 714 server_base.cc:1034] running on GCE node
I20260925 05:00:41.691558 714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:41.692355 714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:41.693548 714 hybrid_clock.cc:648] HybridClock initialized: now 1790312441693506 us; error 63 us; skew 500 ppm
I20260925 05:00:41.693833 714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:41 dist-test-slave-5n86 krb5kdc[320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312441, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:41.696973 714 init.cc:377] Logged in from keytab as kudu/127.30.247.3@KRBTEST.COM (short username kudu)
I20260925 05:00:41.698385 714 webserver.cc:460] Webserver started at http://127.30.247.3:35189/ using document root <none> and password file <none>
I20260925 05:00:41.699023 714 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:41.699105 714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:41.699450 714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:41.702207 714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-2/data/instance:
uuid: "8de2708cdd294f93ae6e26dcf829a870"
format_stamp: "Formatted at 2026-09-25 05:00:41 on dist-test-slave-5n86"
server_key: "df1cca0818026b8cd22e61ec5073bb63"
server_key_iv: "35082b9727565d01a3721475b773b650"
server_key_version: "encryptionkey@0"
I20260925 05:00:41.702958 714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance:
uuid: "8de2708cdd294f93ae6e26dcf829a870"
format_stamp: "Formatted at 2026-09-25 05:00:41 on dist-test-slave-5n86"
server_key: "df1cca0818026b8cd22e61ec5073bb63"
server_key_iv: "35082b9727565d01a3721475b773b650"
server_key_version: "encryptionkey@0"
I20260925 05:00:41.707732 714 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 05:00:41.710836 729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:41.712009 714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 05:00:41.712164 714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "8de2708cdd294f93ae6e26dcf829a870"
format_stamp: "Formatted at 2026-09-25 05:00:41 on dist-test-slave-5n86"
server_key: "df1cca0818026b8cd22e61ec5073bb63"
server_key_iv: "35082b9727565d01a3721475b773b650"
server_key_version: "encryptionkey@0"
I20260925 05:00:41.712369 714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:41.727823 714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:41.731805 714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:41.732064 714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:41.734519 714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:00:41.734625 714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:41.734782 714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:00:41.734905 714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 25 05:00:41 dist-test-slave-5n86 krb5kdc[320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312441, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:00:41.756150 413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:41.734896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:41309 (local address 127.30.247.62:45447)
0925 05:00:41.735057 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:41.735062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:41.736341 (+ 1279us) server_negotiation.cc:407] Connection header received
0925 05:00:41.738440 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:41.738447 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:41.738537 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:41.738670 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:41.741010 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.742106 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.744391 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.744889 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.745017 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:41.749380 (+ 4363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:41.749386 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:41.749389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:41.749419 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:41.752646 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:41.753779 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:41.753784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:41.753788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:41.753944 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:41.754377 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:41.754382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:41.754385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:41.754716 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:41.754877 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:41.755590 (+ 713us) server_negotiation.cc:299] Negotiation successful
0925 05:00:41.755806 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 05:00:41.757421 738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:41.735443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:45447 (local address 127.30.247.3:41309)
0925 05:00:41.736199 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:41.736235 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:41.738148 (+ 1913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:41.738972 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:41.738988 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:41.739840 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:41.740808 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:41.740837 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.742250 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.742255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:41.744240 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:41.744251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.745195 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.745201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:41.745552 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:41.746351 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:41.746387 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:41.749154 (+ 2767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:41.752773 (+ 3619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:41.752786 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:41.752801 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:41.753618 (+ 817us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:41.754057 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:41.754064 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:41.754069 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:41.754259 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:41.754841 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:41.754849 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:41.755219 (+ 370us) client_negotiation.cc:769] Sending connection context
0925 05:00:41.755480 (+ 261us) client_negotiation.cc:240] Negotiation successful
0925 05:00:41.755972 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":342,"threads_started":1}
I20260925 05:00:41.789765 797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:41.776500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:45447 (local address 127.30.247.3:55959)
0925 05:00:41.777120 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:41.777135 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:41.777310 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:41.777651 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:41.777655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:41.778122 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:41.778428 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:41.778437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.779404 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.779410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:41.780389 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:41.780400 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.781472 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.781478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:41.781720 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:41.782411 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:41.782429 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:41.783110 (+ 681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:41.786896 (+ 3786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:41.786903 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:41.786910 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:41.787508 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:41.788055 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:41.788058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:41.788061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:41.788173 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:41.788877 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:41.788888 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:41.789029 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 05:00:41.789178 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 05:00:41.789432 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":339,"threads_started":1}
I20260925 05:00:41.792784 413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:41.776323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:55959 (local address 127.30.247.62:45447)
0925 05:00:41.776522 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:41.776527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:41.777233 (+ 706us) server_negotiation.cc:407] Connection header received
0925 05:00:41.777459 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:41.777463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:41.777523 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:41.777626 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:41.778570 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.779267 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.780590 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.781277 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.781420 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:41.783287 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:41.783295 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:41.783300 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:41.783344 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:41.786701 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:41.787645 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:41.787654 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:41.787660 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:41.787908 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:41.788277 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:41.788284 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:41.788291 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:41.788745 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:41.788910 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:41.789340 (+ 430us) server_negotiation.cc:299] Negotiation successful
0925 05:00:41.789625 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 05:00:41.803651 571 heartbeater.cc:503] Master 127.30.247.62:45447 was elected leader, sending a full tablet report...
I20260925 05:00:41.807016 714 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:34529
I20260925 05:00:41.807020 849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:34529 every 8 connection(s)
I20260925 05:00:41.808200 714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:00:41.810336 850 heartbeater.cc:347] Connected to a master server at 127.30.247.62:45447
I20260925 05:00:41.810576 850 heartbeater.cc:464] Registering TS with master...
I20260925 05:00:41.811241 850 heartbeater.cc:511] Master 127.30.247.62:45447 requested a full tablet report, sending...
I20260925 05:00:41.811558 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 714
I20260925 05:00:41.811790 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance
I20260925 05:00:41.812243 31708 external_mini_cluster.cc:1428] Setting key f536e022322841a6f8044bc67a599149
I20260925 05:00:41.812666 365 ts_manager.cc:194] Registered new tserver with Master: 8de2708cdd294f93ae6e26dcf829a870 (127.30.247.3:34529)
I20260925 05:00:41.813678 365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.3@KRBTEST.COM'} at 127.30.247.3:41309
I20260925 05:00:41.815169 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 05:00:41 dist-test-slave-5n86 krb5kdc[320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312441, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 05:00:41 dist-test-slave-5n86 krb5kdc[320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312441, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:00:41.858500 413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:41.844622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45050 (local address 127.30.247.62:45447)
0925 05:00:41.844859 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:41.844867 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:41.845086 (+ 219us) server_negotiation.cc:407] Connection header received
0925 05:00:41.845316 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:41.845320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:41.845381 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:41.845486 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:41.846805 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.847525 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.848867 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.849287 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.849419 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:41.853046 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:41.853051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:41.853054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:41.853096 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:41.855850 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:41.856505 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:41.856510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:41.856513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:41.856697 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:41.857081 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:41.857086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:41.857089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:41.857366 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:41.857617 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:41.857930 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 05:00:41.858146 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 05:00:41.862886 365 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:00:41.869972 365 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:00:41.906152 867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:41.895982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58000 (local address 127.30.247.2:34935)
0925 05:00:41.897085 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:41.897094 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:41.897914 (+ 820us) server_negotiation.cc:407] Connection header received
0925 05:00:41.898112 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:41.898120 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:41.898331 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:41.898524 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:00:41.899607 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.901123 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.903163 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.904494 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.904656 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:41.904862 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:41.905711 (+ 849us) server_negotiation.cc:299] Negotiation successful
0925 05:00:41.905877 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":963,"thread_start_us":831,"threads_started":1}
I20260925 05:00:41.906459 865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:41.881986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38258 (local address 127.30.247.1:33069)
0925 05:00:41.897507 (+ 15521us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:41.897517 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:41.897548 (+ 31us) server_negotiation.cc:407] Connection header received
0925 05:00:41.897650 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:41.897659 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:41.897922 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:41.898104 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:00:41.899527 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.900755 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.903448 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.904898 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.905024 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:41.905201 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:41.905998 (+ 797us) server_negotiation.cc:299] Negotiation successful
0925 05:00:41.906160 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15358,"thread_start_us":15237,"threads_started":1}
I20260925 05:00:41.906476 866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:41.882287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55034 (local address 127.30.247.3:34529)
0925 05:00:41.896429 (+ 14142us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:41.896440 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:41.897166 (+ 726us) server_negotiation.cc:407] Connection header received
0925 05:00:41.897309 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:41.897322 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:41.897657 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:41.897878 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:00:41.899356 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.901089 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.903676 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.904643 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.904765 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:41.904990 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:41.906020 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0925 05:00:41.906155 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13961,"thread_start_us":13803,"threads_started":1}
I20260925 05:00:41.906733 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:41.881920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:34529 (local address 127.0.0.1:55034)
0925 05:00:41.896789 (+ 14869us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:41.896813 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:41.896963 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:41.898178 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:41.898183 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:41.898212 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:00:41.899104 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:41.899126 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.901224 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.901229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:41.903535 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:41.903545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.905129 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.905146 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:41.905350 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:41.905381 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 05:00:41.905469 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 05:00:41.905610 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14718,"mutex_wait_us":62,"thread_start_us":12671,"threads_started":1}
I20260925 05:00:41.907055 864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:41.881781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:33069 (local address 127.0.0.1:38258)
0925 05:00:41.895121 (+ 13340us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:41.895192 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:41.895420 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:41.898105 (+ 2685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:41.898112 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:41.898140 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:00:41.899333 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:41.899344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.900877 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.900884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:41.903294 (+ 2410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:41.903308 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.905339 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.905344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:41.905514 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:41.905544 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 05:00:41.905617 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 05:00:41.905732 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12975,"thread_start_us":14667,"threads_started":1}
I20260925 05:00:41.907405 868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:41.897123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:34935 (local address 127.0.0.1:58000)
0925 05:00:41.897807 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:41.897822 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:41.897934 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:41.898529 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:41.898558 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:41.898587 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:00:41.899385 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:41.899404 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.901281 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.901284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:41.903015 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:41.903027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:41.904993 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:41.904999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:41.905203 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:41.905283 (+ 80us) client_negotiation.cc:769] Sending connection context
0925 05:00:41.905417 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 05:00:41.905603 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":87,"thread_start_us":433,"threads_started":1}
I20260925 05:00:41.910193 779 tablet_service.cc:1467] Processing CreateTablet for tablet cf09394f7412414d89b0c642ed77a632 (DEFAULT_TABLE table=test-table [id=953713c419d54da886a914867af86e93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:00:41.910534 640 tablet_service.cc:1467] Processing CreateTablet for tablet cf09394f7412414d89b0c642ed77a632 (DEFAULT_TABLE table=test-table [id=953713c419d54da886a914867af86e93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:00:41.910534 501 tablet_service.cc:1467] Processing CreateTablet for tablet cf09394f7412414d89b0c642ed77a632 (DEFAULT_TABLE table=test-table [id=953713c419d54da886a914867af86e93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:00:41.911425 779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf09394f7412414d89b0c642ed77a632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:41.911804 501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf09394f7412414d89b0c642ed77a632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:41.912825 640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf09394f7412414d89b0c642ed77a632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:41.922364 869 tablet_bootstrap.cc:492] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870: Bootstrap starting.
I20260925 05:00:41.925336 870 tablet_bootstrap.cc:492] T cf09394f7412414d89b0c642ed77a632 P 9b3474e03feb4e0f96eb9dc84b4f568c: Bootstrap starting.
I20260925 05:00:41.925957 869 tablet_bootstrap.cc:654] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:41.926637 871 tablet_bootstrap.cc:492] T cf09394f7412414d89b0c642ed77a632 P 7a2800457a6e43d9a8d54a9240e98c1e: Bootstrap starting.
I20260925 05:00:41.927318 869 log.cc:824] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:41.928699 870 tablet_bootstrap.cc:654] T cf09394f7412414d89b0c642ed77a632 P 9b3474e03feb4e0f96eb9dc84b4f568c: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:41.929603 870 log.cc:824] T cf09394f7412414d89b0c642ed77a632 P 9b3474e03feb4e0f96eb9dc84b4f568c: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:41.929695 871 tablet_bootstrap.cc:654] T cf09394f7412414d89b0c642ed77a632 P 7a2800457a6e43d9a8d54a9240e98c1e: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:41.930366 869 tablet_bootstrap.cc:492] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870: No bootstrap required, opened a new log
I20260925 05:00:41.930853 871 log.cc:824] T cf09394f7412414d89b0c642ed77a632 P 7a2800457a6e43d9a8d54a9240e98c1e: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:41.931339 869 ts_tablet_manager.cc:1397] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 05:00:41.932116 870 tablet_bootstrap.cc:492] T cf09394f7412414d89b0c642ed77a632 P 9b3474e03feb4e0f96eb9dc84b4f568c: No bootstrap required, opened a new log
I20260925 05:00:41.932361 870 ts_tablet_manager.cc:1397] T cf09394f7412414d89b0c642ed77a632 P 9b3474e03feb4e0f96eb9dc84b4f568c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260925 05:00:41.938757 870 raft_consensus.cc:348] T cf09394f7412414d89b0c642ed77a632 P 9b3474e03feb4e0f96eb9dc84b4f568c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } }
I20260925 05:00:41.939081 870 raft_consensus.cc:374] T cf09394f7412414d89b0c642ed77a632 P 9b3474e03feb4e0f96eb9dc84b4f568c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:41.939232 870 raft_consensus.cc:729] T cf09394f7412414d89b0c642ed77a632 P 9b3474e03feb4e0f96eb9dc84b4f568c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b3474e03feb4e0f96eb9dc84b4f568c, State: Initialized, Role: FOLLOWER
I20260925 05:00:41.939925 870 consensus_queue.cc:260] T cf09394f7412414d89b0c642ed77a632 P 9b3474e03feb4e0f96eb9dc84b4f568c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } }
I20260925 05:00:41.941560 871 tablet_bootstrap.cc:492] T cf09394f7412414d89b0c642ed77a632 P 7a2800457a6e43d9a8d54a9240e98c1e: No bootstrap required, opened a new log
I20260925 05:00:41.941864 871 ts_tablet_manager.cc:1397] T cf09394f7412414d89b0c642ed77a632 P 7a2800457a6e43d9a8d54a9240e98c1e: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20260925 05:00:41.942268 870 ts_tablet_manager.cc:1428] T cf09394f7412414d89b0c642ed77a632 P 9b3474e03feb4e0f96eb9dc84b4f568c: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20260925 05:00:41.944339 869 raft_consensus.cc:348] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } }
I20260925 05:00:41.944835 869 raft_consensus.cc:374] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:41.944931 869 raft_consensus.cc:729] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de2708cdd294f93ae6e26dcf829a870, State: Initialized, Role: FOLLOWER
I20260925 05:00:41.945493 869 consensus_queue.cc:260] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } }
I20260925 05:00:41.947010 850 heartbeater.cc:503] Master 127.30.247.62:45447 was elected leader, sending a full tablet report...
I20260925 05:00:41.947616 869 ts_tablet_manager.cc:1428] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260925 05:00:41.951149 871 raft_consensus.cc:348] T cf09394f7412414d89b0c642ed77a632 P 7a2800457a6e43d9a8d54a9240e98c1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } }
I20260925 05:00:41.951464 871 raft_consensus.cc:374] T cf09394f7412414d89b0c642ed77a632 P 7a2800457a6e43d9a8d54a9240e98c1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:41.951574 871 raft_consensus.cc:729] T cf09394f7412414d89b0c642ed77a632 P 7a2800457a6e43d9a8d54a9240e98c1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a2800457a6e43d9a8d54a9240e98c1e, State: Initialized, Role: FOLLOWER
I20260925 05:00:41.952052 871 consensus_queue.cc:260] T cf09394f7412414d89b0c642ed77a632 P 7a2800457a6e43d9a8d54a9240e98c1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } }
I20260925 05:00:41.952952 710 heartbeater.cc:503] Master 127.30.247.62:45447 was elected leader, sending a full tablet report...
I20260925 05:00:41.953558 871 ts_tablet_manager.cc:1428] T cf09394f7412414d89b0c642ed77a632 P 7a2800457a6e43d9a8d54a9240e98c1e: Time spent starting tablet: real 0.012s user 0.006s sys 0.004s
W20260925 05:00:42.039585 711 tablet.cc:2369] T cf09394f7412414d89b0c642ed77a632 P 7a2800457a6e43d9a8d54a9240e98c1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:00:42.044180 572 tablet.cc:2369] T cf09394f7412414d89b0c642ed77a632 P 9b3474e03feb4e0f96eb9dc84b4f568c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:00:42.059818 851 tablet.cc:2369] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:00:42.301545 876 raft_consensus.cc:482] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:00:42.301783 876 raft_consensus.cc:504] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } }
I20260925 05:00:42.303333 876 leader_election.cc:288] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b3474e03feb4e0f96eb9dc84b4f568c (127.30.247.1:33069), 7a2800457a6e43d9a8d54a9240e98c1e (127.30.247.2:34935)
I20260925 05:00:42.306159 877 raft_consensus.cc:482] T cf09394f7412414d89b0c642ed77a632 P 7a2800457a6e43d9a8d54a9240e98c1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:00:42.306510 877 raft_consensus.cc:504] T cf09394f7412414d89b0c642ed77a632 P 7a2800457a6e43d9a8d54a9240e98c1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } }
I20260925 05:00:42.311203 867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:42.303698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:55395 (local address 127.30.247.2:34935)
0925 05:00:42.303887 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:42.303893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:42.304506 (+ 613us) server_negotiation.cc:407] Connection header received
0925 05:00:42.304675 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:42.304680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:42.304751 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:42.304859 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:00:42.305373 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:42.306216 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:42.308787 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:42.310077 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:42.310211 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:42.310278 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:42.310814 (+ 536us) server_negotiation.cc:299] Negotiation successful
0925 05:00:42.310943 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 05:00:42.311386 865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:42.303754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:57305 (local address 127.30.247.1:33069)
0925 05:00:42.303941 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:42.303948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:42.304289 (+ 341us) server_negotiation.cc:407] Connection header received
0925 05:00:42.304451 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:42.304460 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:42.304558 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:42.304644 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:00:42.305202 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:42.306052 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:42.308801 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:42.310077 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:42.310205 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:42.310278 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:42.311034 (+ 756us) server_negotiation.cc:299] Negotiation successful
0925 05:00:42.311154 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 05:00:42.311910 878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:42.303617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:34935 (local address 127.30.247.3:55395)
0925 05:00:42.304383 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:42.304401 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:42.304517 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:42.304873 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:42.304876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:42.304898 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:00:42.305203 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:42.305214 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:42.306358 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:42.306363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:42.308610 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:42.308624 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:42.310422 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:42.310429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:42.310628 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:42.310658 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 05:00:42.311410 (+ 752us) client_negotiation.cc:240] Negotiation successful
0925 05:00:42.311546 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":24,"thread_start_us":309,"threads_started":1}
I20260925 05:00:42.312659 879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:42.303554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:33069 (local address 127.30.247.3:57305)
0925 05:00:42.304187 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:42.304202 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:42.304311 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:42.304674 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:42.304680 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:42.304715 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:00:42.305034 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:42.305056 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:42.306186 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:42.306191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:42.308610 (+ 2419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:42.308627 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:42.310464 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:42.310469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:42.310871 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:42.310903 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 05:00:42.311610 (+ 707us) client_negotiation.cc:240] Negotiation successful
0925 05:00:42.311743 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":125,"thread_start_us":485,"threads_started":1}
I20260925 05:00:42.312937 877 leader_election.cc:288] T cf09394f7412414d89b0c642ed77a632 P 7a2800457a6e43d9a8d54a9240e98c1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b3474e03feb4e0f96eb9dc84b4f568c (127.30.247.1:33069), 8de2708cdd294f93ae6e26dcf829a870 (127.30.247.3:34529)
I20260925 05:00:42.312971 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf09394f7412414d89b0c642ed77a632" candidate_uuid: "8de2708cdd294f93ae6e26dcf829a870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" is_pre_election: true
I20260925 05:00:42.313616 664 raft_consensus.cc:2454] T cf09394f7412414d89b0c642ed77a632 P 7a2800457a6e43d9a8d54a9240e98c1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de2708cdd294f93ae6e26dcf829a870 in term 0.
I20260925 05:00:42.314436 731 leader_election.cc:302] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a2800457a6e43d9a8d54a9240e98c1e, 8de2708cdd294f93ae6e26dcf829a870; no voters:
I20260925 05:00:42.314864 876 raft_consensus.cc:2781] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:00:42.315018 876 raft_consensus.cc:482] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:00:42.315155 876 raft_consensus.cc:3037] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:42.315109 526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf09394f7412414d89b0c642ed77a632" candidate_uuid: "8de2708cdd294f93ae6e26dcf829a870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" is_pre_election: true
I20260925 05:00:42.315636 526 raft_consensus.cc:2454] T cf09394f7412414d89b0c642ed77a632 P 9b3474e03feb4e0f96eb9dc84b4f568c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de2708cdd294f93ae6e26dcf829a870 in term 0.
I20260925 05:00:42.318260 876 raft_consensus.cc:504] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } }
I20260925 05:00:42.319274 876 leader_election.cc:288] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870 [CANDIDATE]: Term 1 election: Requested vote from peers 9b3474e03feb4e0f96eb9dc84b4f568c (127.30.247.1:33069), 7a2800457a6e43d9a8d54a9240e98c1e (127.30.247.2:34935)
I20260925 05:00:42.319253 526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf09394f7412414d89b0c642ed77a632" candidate_uuid: "8de2708cdd294f93ae6e26dcf829a870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c"
I20260925 05:00:42.319463 526 raft_consensus.cc:3037] T cf09394f7412414d89b0c642ed77a632 P 9b3474e03feb4e0f96eb9dc84b4f568c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:42.319577 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf09394f7412414d89b0c642ed77a632" candidate_uuid: "8de2708cdd294f93ae6e26dcf829a870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a2800457a6e43d9a8d54a9240e98c1e"
I20260925 05:00:42.319753 664 raft_consensus.cc:3037] T cf09394f7412414d89b0c642ed77a632 P 7a2800457a6e43d9a8d54a9240e98c1e [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:42.320938 880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:42.308434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:33069 (local address 127.30.247.2:36487)
0925 05:00:42.314246 (+ 5812us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:42.314262 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:42.314410 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:42.314836 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:42.314840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:42.314868 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:00:42.315414 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:42.315431 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:42.316983 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:42.316988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:42.318669 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:42.318680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:42.320143 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:42.320150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:42.320327 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:42.320355 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 05:00:42.320456 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 05:00:42.320595 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5701,"thread_start_us":3692,"threads_started":1}
I20260925 05:00:42.320946 865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:42.309070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:36487 (local address 127.30.247.1:33069)
0925 05:00:42.311576 (+ 2506us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:42.311581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:42.314328 (+ 2747us) server_negotiation.cc:407] Connection header received
0925 05:00:42.314536 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:42.314541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:42.314602 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:42.314653 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:00:42.315657 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:42.316823 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:42.318803 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:42.319894 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:42.320071 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:42.320145 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:42.320482 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 05:00:42.320643 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2419,"thread_start_us":4611,"threads_started":1}
I20260925 05:00:42.322835 526 raft_consensus.cc:2454] T cf09394f7412414d89b0c642ed77a632 P 9b3474e03feb4e0f96eb9dc84b4f568c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de2708cdd294f93ae6e26dcf829a870 in term 1.
I20260925 05:00:42.322881 664 raft_consensus.cc:2454] T cf09394f7412414d89b0c642ed77a632 P 7a2800457a6e43d9a8d54a9240e98c1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de2708cdd294f93ae6e26dcf829a870 in term 1.
I20260925 05:00:42.323199 731 leader_election.cc:302] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a2800457a6e43d9a8d54a9240e98c1e, 8de2708cdd294f93ae6e26dcf829a870; no voters:
I20260925 05:00:42.323503 876 raft_consensus.cc:2781] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:00:42.323625 525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf09394f7412414d89b0c642ed77a632" candidate_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" is_pre_election: true
I20260925 05:00:42.323935 876 raft_consensus.cc:686] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870 [term 1 LEADER]: Becoming Leader. State: Replica: 8de2708cdd294f93ae6e26dcf829a870, State: Running, Role: LEADER
I20260925 05:00:42.324049 525 raft_consensus.cc:2379] T cf09394f7412414d89b0c642ed77a632 P 9b3474e03feb4e0f96eb9dc84b4f568c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a2800457a6e43d9a8d54a9240e98c1e in current term 1: Already voted for candidate 8de2708cdd294f93ae6e26dcf829a870 in this term.
I20260925 05:00:42.324466 876 consensus_queue.cc:237] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } }
I20260925 05:00:42.325157 866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:42.309401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:50647 (local address 127.30.247.3:34529)
0925 05:00:42.309629 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:42.309637 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:42.312467 (+ 2830us) server_negotiation.cc:407] Connection header received
0925 05:00:42.312751 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:42.312756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:42.312823 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:42.312888 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:00:42.313778 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:42.314820 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:42.317379 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:42.318687 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:42.318853 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:42.318934 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:42.324773 (+ 5839us) server_negotiation.cc:299] Negotiation successful
0925 05:00:42.324891 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 05:00:42.325172 881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:42.308388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:34529 (local address 127.30.247.2:50647)
0925 05:00:42.312378 (+ 3990us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:42.312401 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:42.312582 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:42.313078 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:42.313085 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:42.313117 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:00:42.313601 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:42.313625 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:42.314997 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:42.315001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:42.317237 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:42.317250 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:42.319075 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:42.319082 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:42.323355 (+ 4273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:42.324635 (+ 1280us) client_negotiation.cc:769] Sending connection context
0925 05:00:42.324738 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 05:00:42.324861 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3842,"thread_start_us":5604,"threads_started":1}
I20260925 05:00:42.327070 805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf09394f7412414d89b0c642ed77a632" candidate_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de2708cdd294f93ae6e26dcf829a870" is_pre_election: true
I20260925 05:00:42.328006 592 leader_election.cc:302] T cf09394f7412414d89b0c642ed77a632 P 7a2800457a6e43d9a8d54a9240e98c1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a2800457a6e43d9a8d54a9240e98c1e; no voters: 8de2708cdd294f93ae6e26dcf829a870, 9b3474e03feb4e0f96eb9dc84b4f568c
I20260925 05:00:42.328375 877 raft_consensus.cc:2726] T cf09394f7412414d89b0c642ed77a632 P 7a2800457a6e43d9a8d54a9240e98c1e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 05:00:42.329310 365 catalog_manager.cc:5495] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8de2708cdd294f93ae6e26dcf829a870 (127.30.247.3). New cstate: current_term: 1 leader_uuid: "8de2708cdd294f93ae6e26dcf829a870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } health_report { overall_health: UNKNOWN } } }
I20260925 05:00:42.349903 866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:42.344600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55042 (local address 127.30.247.3:34529)
0925 05:00:42.344797 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:42.344802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:42.344842 (+ 40us) server_negotiation.cc:407] Connection header received
0925 05:00:42.345140 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:42.345144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:42.345247 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:42.345377 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:00:42.345685 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:42.346427 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:42.347739 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:42.348134 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:42.348238 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:42.348789 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:00:42.349269 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:00:42.349378 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:42.349516 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 05:00:42.349588 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 05:00:42.357635 526 raft_consensus.cc:1261] T cf09394f7412414d89b0c642ed77a632 P 9b3474e03feb4e0f96eb9dc84b4f568c [term 1 FOLLOWER]: Refusing update from remote peer 8de2708cdd294f93ae6e26dcf829a870: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:00:42.357671 664 raft_consensus.cc:1261] T cf09394f7412414d89b0c642ed77a632 P 7a2800457a6e43d9a8d54a9240e98c1e [term 1 FOLLOWER]: Refusing update from remote peer 8de2708cdd294f93ae6e26dcf829a870: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:00:42.358608 876 consensus_queue.cc:1035] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:00:42.358978 883 consensus_queue.cc:1035] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:00:42.374396 887 mvcc.cc:205] Tried to move back new op lower bound from 7333119763887419392 to 7333119763765350400. Current Snapshot: MvccSnapshot[applied={T|T < 7333119763887419392}]
I20260925 05:00:42.388670 365 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:00:42.388940 365 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:00:42.392251 365 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e065cd42faf44fe08e2635e69533842a: Sending DeleteTablet for 3 replicas of tablet cf09394f7412414d89b0c642ed77a632
I20260925 05:00:42.393392 501 tablet_service.cc:1514] Processing DeleteTablet for tablet cf09394f7412414d89b0c642ed77a632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:00:42 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:38258
I20260925 05:00:42.393512 779 tablet_service.cc:1514] Processing DeleteTablet for tablet cf09394f7412414d89b0c642ed77a632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:00:42 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:55034
I20260925 05:00:42.394397 640 tablet_service.cc:1514] Processing DeleteTablet for tablet cf09394f7412414d89b0c642ed77a632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:00:42 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:58000
I20260925 05:00:42.394595 364 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:00:42.395344 364 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:00:42.397070 897 tablet_replica.cc:330] T cf09394f7412414d89b0c642ed77a632 P 9b3474e03feb4e0f96eb9dc84b4f568c: stopping tablet replica
I20260925 05:00:42.397548 897 raft_consensus.cc:2229] T cf09394f7412414d89b0c642ed77a632 P 9b3474e03feb4e0f96eb9dc84b4f568c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:00:42.398140 897 raft_consensus.cc:2258] T cf09394f7412414d89b0c642ed77a632 P 9b3474e03feb4e0f96eb9dc84b4f568c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:00:42.401835 897 ts_tablet_manager.cc:1905] T cf09394f7412414d89b0c642ed77a632 P 9b3474e03feb4e0f96eb9dc84b4f568c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:00:42.403515 500 tablet_service.cc:1467] Processing CreateTablet for tablet bcbf561275fe4e3788764bdc17708a3f (DEFAULT_TABLE table=test-table [id=11ebce7973754b1ca017769e64e706fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:00:42.403965 500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcbf561275fe4e3788764bdc17708a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:42.403983 778 tablet_service.cc:1467] Processing CreateTablet for tablet bcbf561275fe4e3788764bdc17708a3f (DEFAULT_TABLE table=test-table [id=11ebce7973754b1ca017769e64e706fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:00:42.404572 778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcbf561275fe4e3788764bdc17708a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:42.407392 899 tablet_replica.cc:330] T cf09394f7412414d89b0c642ed77a632 P 7a2800457a6e43d9a8d54a9240e98c1e: stopping tablet replica
I20260925 05:00:42.407563 639 tablet_service.cc:1467] Processing CreateTablet for tablet bcbf561275fe4e3788764bdc17708a3f (DEFAULT_TABLE table=test-table [id=11ebce7973754b1ca017769e64e706fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:00:42.407938 899 raft_consensus.cc:2229] T cf09394f7412414d89b0c642ed77a632 P 7a2800457a6e43d9a8d54a9240e98c1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:00:42.408103 639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcbf561275fe4e3788764bdc17708a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:42.408591 899 raft_consensus.cc:2258] T cf09394f7412414d89b0c642ed77a632 P 7a2800457a6e43d9a8d54a9240e98c1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:00:42.411700 870 tablet_bootstrap.cc:492] T bcbf561275fe4e3788764bdc17708a3f P 9b3474e03feb4e0f96eb9dc84b4f568c: Bootstrap starting.
I20260925 05:00:42.414062 870 tablet_bootstrap.cc:654] T bcbf561275fe4e3788764bdc17708a3f P 9b3474e03feb4e0f96eb9dc84b4f568c: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:42.415297 871 tablet_bootstrap.cc:492] T bcbf561275fe4e3788764bdc17708a3f P 7a2800457a6e43d9a8d54a9240e98c1e: Bootstrap starting.
I20260925 05:00:42.416971 897 ts_tablet_manager.cc:1918] T cf09394f7412414d89b0c642ed77a632 P 9b3474e03feb4e0f96eb9dc84b4f568c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:00:42.417070 871 tablet_bootstrap.cc:654] T bcbf561275fe4e3788764bdc17708a3f P 7a2800457a6e43d9a8d54a9240e98c1e: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:42.417182 897 log.cc:1196] T cf09394f7412414d89b0c642ed77a632 P 9b3474e03feb4e0f96eb9dc84b4f568c: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-0/wal/wals/cf09394f7412414d89b0c642ed77a632
I20260925 05:00:42.417742 897 ts_tablet_manager.cc:1939] T cf09394f7412414d89b0c642ed77a632 P 9b3474e03feb4e0f96eb9dc84b4f568c: Deleting consensus metadata
I20260925 05:00:42.418072 869 tablet_bootstrap.cc:492] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870: Bootstrap starting.
I20260925 05:00:42.419612 351 catalog_manager.cc:4841] TS 9b3474e03feb4e0f96eb9dc84b4f568c (127.30.247.1:33069): tablet cf09394f7412414d89b0c642ed77a632 (table test-table [id=953713c419d54da886a914867af86e93]) successfully deleted
I20260925 05:00:42.421171 869 tablet_bootstrap.cc:654] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:42.421391 899 ts_tablet_manager.cc:1905] T cf09394f7412414d89b0c642ed77a632 P 7a2800457a6e43d9a8d54a9240e98c1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:00:42.424389 898 tablet_replica.cc:330] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870: stopping tablet replica
I20260925 05:00:42.427894 871 tablet_bootstrap.cc:492] T bcbf561275fe4e3788764bdc17708a3f P 7a2800457a6e43d9a8d54a9240e98c1e: No bootstrap required, opened a new log
I20260925 05:00:42.428061 871 ts_tablet_manager.cc:1397] T bcbf561275fe4e3788764bdc17708a3f P 7a2800457a6e43d9a8d54a9240e98c1e: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.002s
I20260925 05:00:42.429018 871 raft_consensus.cc:348] T bcbf561275fe4e3788764bdc17708a3f P 7a2800457a6e43d9a8d54a9240e98c1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } }
I20260925 05:00:42.429253 871 raft_consensus.cc:374] T bcbf561275fe4e3788764bdc17708a3f P 7a2800457a6e43d9a8d54a9240e98c1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:42.429363 871 raft_consensus.cc:729] T bcbf561275fe4e3788764bdc17708a3f P 7a2800457a6e43d9a8d54a9240e98c1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a2800457a6e43d9a8d54a9240e98c1e, State: Initialized, Role: FOLLOWER
I20260925 05:00:42.429666 870 tablet_bootstrap.cc:492] T bcbf561275fe4e3788764bdc17708a3f P 9b3474e03feb4e0f96eb9dc84b4f568c: No bootstrap required, opened a new log
I20260925 05:00:42.429687 869 tablet_bootstrap.cc:492] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870: No bootstrap required, opened a new log
I20260925 05:00:42.429808 869 ts_tablet_manager.cc:1397] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20260925 05:00:42.429663 871 consensus_queue.cc:260] T bcbf561275fe4e3788764bdc17708a3f P 7a2800457a6e43d9a8d54a9240e98c1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } }
I20260925 05:00:42.429924 870 ts_tablet_manager.cc:1397] T bcbf561275fe4e3788764bdc17708a3f P 9b3474e03feb4e0f96eb9dc84b4f568c: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.003s
I20260925 05:00:42.430228 899 ts_tablet_manager.cc:1918] T cf09394f7412414d89b0c642ed77a632 P 7a2800457a6e43d9a8d54a9240e98c1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:00:42.430400 899 log.cc:1196] T cf09394f7412414d89b0c642ed77a632 P 7a2800457a6e43d9a8d54a9240e98c1e: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-1/wal/wals/cf09394f7412414d89b0c642ed77a632
I20260925 05:00:42.430554 869 raft_consensus.cc:348] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } }
I20260925 05:00:42.430797 869 raft_consensus.cc:374] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:42.430907 869 raft_consensus.cc:729] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de2708cdd294f93ae6e26dcf829a870, State: Initialized, Role: FOLLOWER
I20260925 05:00:42.430938 899 ts_tablet_manager.cc:1939] T cf09394f7412414d89b0c642ed77a632 P 7a2800457a6e43d9a8d54a9240e98c1e: Deleting consensus metadata
I20260925 05:00:42.431213 871 ts_tablet_manager.cc:1428] T bcbf561275fe4e3788764bdc17708a3f P 7a2800457a6e43d9a8d54a9240e98c1e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:00:42.431131 870 raft_consensus.cc:348] T bcbf561275fe4e3788764bdc17708a3f P 9b3474e03feb4e0f96eb9dc84b4f568c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } }
I20260925 05:00:42.431456 870 raft_consensus.cc:374] T bcbf561275fe4e3788764bdc17708a3f P 9b3474e03feb4e0f96eb9dc84b4f568c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:42.431597 870 raft_consensus.cc:729] T bcbf561275fe4e3788764bdc17708a3f P 9b3474e03feb4e0f96eb9dc84b4f568c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b3474e03feb4e0f96eb9dc84b4f568c, State: Initialized, Role: FOLLOWER
I20260925 05:00:42.431790 898 raft_consensus.cc:2229] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:00:42.431954 870 consensus_queue.cc:260] T bcbf561275fe4e3788764bdc17708a3f P 9b3474e03feb4e0f96eb9dc84b4f568c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } }
I20260925 05:00:42.432410 898 raft_consensus.cc:2258] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:00:42.432822 350 catalog_manager.cc:4841] TS 7a2800457a6e43d9a8d54a9240e98c1e (127.30.247.2:34935): tablet cf09394f7412414d89b0c642ed77a632 (table test-table [id=953713c419d54da886a914867af86e93]) successfully deleted
I20260925 05:00:42.433307 870 ts_tablet_manager.cc:1428] T bcbf561275fe4e3788764bdc17708a3f P 9b3474e03feb4e0f96eb9dc84b4f568c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260925 05:00:42.433071 869 consensus_queue.cc:260] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } }
I20260925 05:00:42.434939 898 ts_tablet_manager.cc:1905] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:00:42.434996 869 ts_tablet_manager.cc:1428] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260925 05:00:42.441332 898 ts_tablet_manager.cc:1918] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:00:42.441502 898 log.cc:1196] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-2/wal/wals/cf09394f7412414d89b0c642ed77a632
I20260925 05:00:42.442001 898 ts_tablet_manager.cc:1939] T cf09394f7412414d89b0c642ed77a632 P 8de2708cdd294f93ae6e26dcf829a870: Deleting consensus metadata
I20260925 05:00:42.443432 350 catalog_manager.cc:4841] TS 8de2708cdd294f93ae6e26dcf829a870 (127.30.247.3:34529): tablet cf09394f7412414d89b0c642ed77a632 (table test-table [id=953713c419d54da886a914867af86e93]) successfully deleted
I20260925 05:00:42.691351 876 raft_consensus.cc:482] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:00:42.691591 876 raft_consensus.cc:504] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } }
I20260925 05:00:42.692394 876 leader_election.cc:288] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b3474e03feb4e0f96eb9dc84b4f568c (127.30.247.1:33069), 7a2800457a6e43d9a8d54a9240e98c1e (127.30.247.2:34935)
I20260925 05:00:42.693002 526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbf561275fe4e3788764bdc17708a3f" candidate_uuid: "8de2708cdd294f93ae6e26dcf829a870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" is_pre_election: true
I20260925 05:00:42.693044 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbf561275fe4e3788764bdc17708a3f" candidate_uuid: "8de2708cdd294f93ae6e26dcf829a870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" is_pre_election: true
I20260925 05:00:42.693267 526 raft_consensus.cc:2454] T bcbf561275fe4e3788764bdc17708a3f P 9b3474e03feb4e0f96eb9dc84b4f568c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de2708cdd294f93ae6e26dcf829a870 in term 0.
I20260925 05:00:42.693318 664 raft_consensus.cc:2454] T bcbf561275fe4e3788764bdc17708a3f P 7a2800457a6e43d9a8d54a9240e98c1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de2708cdd294f93ae6e26dcf829a870 in term 0.
I20260925 05:00:42.693744 732 leader_election.cc:302] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8de2708cdd294f93ae6e26dcf829a870, 9b3474e03feb4e0f96eb9dc84b4f568c; no voters:
I20260925 05:00:42.694053 876 raft_consensus.cc:2781] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:00:42.694199 876 raft_consensus.cc:482] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:00:42.694319 876 raft_consensus.cc:3037] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:42.696421 876 raft_consensus.cc:504] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } }
I20260925 05:00:42.696978 876 leader_election.cc:288] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870 [CANDIDATE]: Term 1 election: Requested vote from peers 9b3474e03feb4e0f96eb9dc84b4f568c (127.30.247.1:33069), 7a2800457a6e43d9a8d54a9240e98c1e (127.30.247.2:34935)
I20260925 05:00:42.697448 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbf561275fe4e3788764bdc17708a3f" candidate_uuid: "8de2708cdd294f93ae6e26dcf829a870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a2800457a6e43d9a8d54a9240e98c1e"
I20260925 05:00:42.697472 526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbf561275fe4e3788764bdc17708a3f" candidate_uuid: "8de2708cdd294f93ae6e26dcf829a870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c"
I20260925 05:00:42.697649 664 raft_consensus.cc:3037] T bcbf561275fe4e3788764bdc17708a3f P 7a2800457a6e43d9a8d54a9240e98c1e [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:42.697659 526 raft_consensus.cc:3037] T bcbf561275fe4e3788764bdc17708a3f P 9b3474e03feb4e0f96eb9dc84b4f568c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:42.699867 664 raft_consensus.cc:2454] T bcbf561275fe4e3788764bdc17708a3f P 7a2800457a6e43d9a8d54a9240e98c1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de2708cdd294f93ae6e26dcf829a870 in term 1.
I20260925 05:00:42.700286 731 leader_election.cc:302] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a2800457a6e43d9a8d54a9240e98c1e, 8de2708cdd294f93ae6e26dcf829a870; no voters:
I20260925 05:00:42.700579 876 raft_consensus.cc:2781] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:00:42.700799 876 raft_consensus.cc:686] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870 [term 1 LEADER]: Becoming Leader. State: Replica: 8de2708cdd294f93ae6e26dcf829a870, State: Running, Role: LEADER
I20260925 05:00:42.700788 526 raft_consensus.cc:2454] T bcbf561275fe4e3788764bdc17708a3f P 9b3474e03feb4e0f96eb9dc84b4f568c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de2708cdd294f93ae6e26dcf829a870 in term 1.
I20260925 05:00:42.701081 876 consensus_queue.cc:237] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } }
I20260925 05:00:42.704056 364 catalog_manager.cc:5495] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8de2708cdd294f93ae6e26dcf829a870 (127.30.247.3). New cstate: current_term: 1 leader_uuid: "8de2708cdd294f93ae6e26dcf829a870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } health_report { overall_health: HEALTHY } } }
Sep 25 05:00:42 dist-test-slave-5n86 krb5kdc[320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312440, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:00:42.821425 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:42.806852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:45447 (local address 127.0.0.1:45060)
0925 05:00:42.807716 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:42.807767 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:42.807910 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:42.808260 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:42.808263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:42.808951 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:42.809152 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:42.809161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:42.810644 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:42.810648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:42.811745 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:42.811757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:42.812712 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:42.812717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:42.812848 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:42.813597 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:42.813614 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:42.815790 (+ 2176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:42.818924 (+ 3134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:42.818934 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:42.818942 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:42.819329 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:42.819684 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:42.819691 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:42.819694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:42.819839 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:42.820312 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:42.820319 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:42.820479 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 05:00:42.820719 (+ 240us) client_negotiation.cc:240] Negotiation successful
0925 05:00:42.820990 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":544,"threads_started":1}
I20260925 05:00:42.821480 909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:42.806835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45060 (local address 127.30.247.62:45447)
0925 05:00:42.807474 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:42.807482 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:42.807860 (+ 378us) server_negotiation.cc:407] Connection header received
0925 05:00:42.808016 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:42.808043 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:42.808115 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:42.808229 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:42.809297 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:42.810475 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:42.811887 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:42.812530 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:42.812667 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:42.815986 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:42.815991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:42.815994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:42.816019 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:42.818790 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:42.819443 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:42.819451 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:42.819455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:42.819582 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:42.819939 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:42.819946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:42.819949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:42.820181 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:42.820344 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:42.820801 (+ 457us) server_negotiation.cc:299] Negotiation successful
0925 05:00:42.821103 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":133,"server-negotiator.queue_time_us":447,"thread_start_us":291,"threads_started":1}
I20260925 05:00:42.826390 364 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:45060:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 05:00:42.827382 364 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:00:42.833673 500 tablet_service.cc:1467] Processing CreateTablet for tablet 6cb95a496bf7481bb127f677bcd69e0a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e2e2735d0d648ae9d472fe685833e42]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:00:42.833807 639 tablet_service.cc:1467] Processing CreateTablet for tablet 6cb95a496bf7481bb127f677bcd69e0a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e2e2735d0d648ae9d472fe685833e42]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:00:42.833973 779 tablet_service.cc:1467] Processing CreateTablet for tablet 6cb95a496bf7481bb127f677bcd69e0a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e2e2735d0d648ae9d472fe685833e42]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:00:42.834197 500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cb95a496bf7481bb127f677bcd69e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:42.834252 639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cb95a496bf7481bb127f677bcd69e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:42.834425 779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cb95a496bf7481bb127f677bcd69e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:42.839633 871 tablet_bootstrap.cc:492] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e: Bootstrap starting.
I20260925 05:00:42.840907 871 tablet_bootstrap.cc:654] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:42.841114 870 tablet_bootstrap.cc:492] T 6cb95a496bf7481bb127f677bcd69e0a P 9b3474e03feb4e0f96eb9dc84b4f568c: Bootstrap starting.
I20260925 05:00:42.841130 869 tablet_bootstrap.cc:492] T 6cb95a496bf7481bb127f677bcd69e0a P 8de2708cdd294f93ae6e26dcf829a870: Bootstrap starting.
I20260925 05:00:42.842793 870 tablet_bootstrap.cc:654] T 6cb95a496bf7481bb127f677bcd69e0a P 9b3474e03feb4e0f96eb9dc84b4f568c: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:42.842837 869 tablet_bootstrap.cc:654] T 6cb95a496bf7481bb127f677bcd69e0a P 8de2708cdd294f93ae6e26dcf829a870: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:42.843298 871 tablet_bootstrap.cc:492] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e: No bootstrap required, opened a new log
I20260925 05:00:42.843477 871 ts_tablet_manager.cc:1397] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 05:00:42.844370 871 raft_consensus.cc:348] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } }
I20260925 05:00:42.844619 871 raft_consensus.cc:374] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:42.844720 871 raft_consensus.cc:729] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a2800457a6e43d9a8d54a9240e98c1e, State: Initialized, Role: FOLLOWER
I20260925 05:00:42.844748 870 tablet_bootstrap.cc:492] T 6cb95a496bf7481bb127f677bcd69e0a P 9b3474e03feb4e0f96eb9dc84b4f568c: No bootstrap required, opened a new log
I20260925 05:00:42.844897 870 ts_tablet_manager.cc:1397] T 6cb95a496bf7481bb127f677bcd69e0a P 9b3474e03feb4e0f96eb9dc84b4f568c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260925 05:00:42.845003 871 consensus_queue.cc:260] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } }
I20260925 05:00:42.845422 877 tablet_replica.cc:439] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } } }
I20260925 05:00:42.845690 877 tablet_replica.cc:442] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:00:42.845567 870 raft_consensus.cc:348] T 6cb95a496bf7481bb127f677bcd69e0a P 9b3474e03feb4e0f96eb9dc84b4f568c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } }
I20260925 05:00:42.845732 869 tablet_bootstrap.cc:492] T 6cb95a496bf7481bb127f677bcd69e0a P 8de2708cdd294f93ae6e26dcf829a870: No bootstrap required, opened a new log
I20260925 05:00:42.845798 870 raft_consensus.cc:374] T 6cb95a496bf7481bb127f677bcd69e0a P 9b3474e03feb4e0f96eb9dc84b4f568c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:42.845867 869 ts_tablet_manager.cc:1397] T 6cb95a496bf7481bb127f677bcd69e0a P 8de2708cdd294f93ae6e26dcf829a870: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 05:00:42.845899 870 raft_consensus.cc:729] T 6cb95a496bf7481bb127f677bcd69e0a P 9b3474e03feb4e0f96eb9dc84b4f568c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b3474e03feb4e0f96eb9dc84b4f568c, State: Initialized, Role: FOLLOWER
I20260925 05:00:42.846180 870 consensus_queue.cc:260] T 6cb95a496bf7481bb127f677bcd69e0a P 9b3474e03feb4e0f96eb9dc84b4f568c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } }
I20260925 05:00:42.846577 869 raft_consensus.cc:348] T 6cb95a496bf7481bb127f677bcd69e0a P 8de2708cdd294f93ae6e26dcf829a870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } }
I20260925 05:00:42.846596 875 tablet_replica.cc:439] T 6cb95a496bf7481bb127f677bcd69e0a P 9b3474e03feb4e0f96eb9dc84b4f568c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } } }
I20260925 05:00:42.846796 869 raft_consensus.cc:374] T 6cb95a496bf7481bb127f677bcd69e0a P 8de2708cdd294f93ae6e26dcf829a870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:42.846873 875 tablet_replica.cc:442] T 6cb95a496bf7481bb127f677bcd69e0a P 9b3474e03feb4e0f96eb9dc84b4f568c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:00:42.846909 869 raft_consensus.cc:729] T 6cb95a496bf7481bb127f677bcd69e0a P 8de2708cdd294f93ae6e26dcf829a870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de2708cdd294f93ae6e26dcf829a870, State: Initialized, Role: FOLLOWER
I20260925 05:00:42.847172 869 consensus_queue.cc:260] T 6cb95a496bf7481bb127f677bcd69e0a P 8de2708cdd294f93ae6e26dcf829a870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } }
I20260925 05:00:42.847570 876 tablet_replica.cc:439] T 6cb95a496bf7481bb127f677bcd69e0a P 8de2708cdd294f93ae6e26dcf829a870: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } } }
I20260925 05:00:42.847841 876 tablet_replica.cc:442] T 6cb95a496bf7481bb127f677bcd69e0a P 8de2708cdd294f93ae6e26dcf829a870: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:00:42.848213 871 ts_tablet_manager.cc:1428] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260925 05:00:42.848712 869 ts_tablet_manager.cc:1428] T 6cb95a496bf7481bb127f677bcd69e0a P 8de2708cdd294f93ae6e26dcf829a870: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 05:00:42.849140 870 ts_tablet_manager.cc:1428] T 6cb95a496bf7481bb127f677bcd69e0a P 9b3474e03feb4e0f96eb9dc84b4f568c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260925 05:00:43.129402 876 consensus_queue.cc:1035] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:00:43.131345 877 raft_consensus.cc:482] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:00:43.131524 877 raft_consensus.cc:504] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } }
I20260925 05:00:43.132174 877 leader_election.cc:288] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b3474e03feb4e0f96eb9dc84b4f568c (127.30.247.1:33069), 8de2708cdd294f93ae6e26dcf829a870 (127.30.247.3:34529)
I20260925 05:00:43.132666 526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb95a496bf7481bb127f677bcd69e0a" candidate_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" is_pre_election: true
I20260925 05:00:43.132917 526 raft_consensus.cc:2454] T 6cb95a496bf7481bb127f677bcd69e0a P 9b3474e03feb4e0f96eb9dc84b4f568c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a2800457a6e43d9a8d54a9240e98c1e in term 0.
I20260925 05:00:43.132943 805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb95a496bf7481bb127f677bcd69e0a" candidate_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de2708cdd294f93ae6e26dcf829a870" is_pre_election: true
I20260925 05:00:43.133219 805 raft_consensus.cc:2454] T 6cb95a496bf7481bb127f677bcd69e0a P 8de2708cdd294f93ae6e26dcf829a870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a2800457a6e43d9a8d54a9240e98c1e in term 0.
I20260925 05:00:43.133726 592 leader_election.cc:302] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7a2800457a6e43d9a8d54a9240e98c1e, 8de2708cdd294f93ae6e26dcf829a870, 9b3474e03feb4e0f96eb9dc84b4f568c; no voters:
I20260925 05:00:43.134106 877 raft_consensus.cc:2781] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:00:43.134238 877 raft_consensus.cc:482] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:00:43.134346 877 raft_consensus.cc:3037] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:43.134975 894 consensus_queue.cc:1035] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:00:43.137477 877 raft_consensus.cc:504] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } }
I20260925 05:00:43.138146 877 leader_election.cc:288] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e [CANDIDATE]: Term 1 election: Requested vote from peers 9b3474e03feb4e0f96eb9dc84b4f568c (127.30.247.1:33069), 8de2708cdd294f93ae6e26dcf829a870 (127.30.247.3:34529)
I20260925 05:00:43.138514 525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb95a496bf7481bb127f677bcd69e0a" candidate_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c"
I20260925 05:00:43.138614 805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb95a496bf7481bb127f677bcd69e0a" candidate_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de2708cdd294f93ae6e26dcf829a870"
I20260925 05:00:43.138722 525 raft_consensus.cc:3037] T 6cb95a496bf7481bb127f677bcd69e0a P 9b3474e03feb4e0f96eb9dc84b4f568c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:43.138833 805 raft_consensus.cc:3037] T 6cb95a496bf7481bb127f677bcd69e0a P 8de2708cdd294f93ae6e26dcf829a870 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:43.141199 525 raft_consensus.cc:2454] T 6cb95a496bf7481bb127f677bcd69e0a P 9b3474e03feb4e0f96eb9dc84b4f568c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a2800457a6e43d9a8d54a9240e98c1e in term 1.
I20260925 05:00:43.141613 805 raft_consensus.cc:2454] T 6cb95a496bf7481bb127f677bcd69e0a P 8de2708cdd294f93ae6e26dcf829a870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a2800457a6e43d9a8d54a9240e98c1e in term 1.
I20260925 05:00:43.141590 593 leader_election.cc:302] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a2800457a6e43d9a8d54a9240e98c1e, 9b3474e03feb4e0f96eb9dc84b4f568c; no voters:
I20260925 05:00:43.141896 877 raft_consensus.cc:2781] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:00:43.142062 877 raft_consensus.cc:686] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e [term 1 LEADER]: Becoming Leader. State: Replica: 7a2800457a6e43d9a8d54a9240e98c1e, State: Running, Role: LEADER
I20260925 05:00:43.142503 877 consensus_queue.cc:237] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } }
I20260925 05:00:43.144243 917 tablet_replica.cc:439] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e: TxnStatusTablet state changed. Reason: New leader 7a2800457a6e43d9a8d54a9240e98c1e. Latest consensus state: current_term: 1 leader_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } } }
I20260925 05:00:43.144555 917 tablet_replica.cc:442] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:00:43.145080 921 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 05:00:43.145328 921 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 05:00:43.145802 365 catalog_manager.cc:5495] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a2800457a6e43d9a8d54a9240e98c1e (127.30.247.2). New cstate: current_term: 1 leader_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } health_report { overall_health: UNKNOWN } } }
I20260925 05:00:43.210819 924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:43.204111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58014 (local address 127.30.247.2:34935)
0925 05:00:43.204971 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:43.204975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:43.204993 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:00:43.205067 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:43.205071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:43.205139 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:43.205262 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:00:43.205674 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:43.206497 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:43.208422 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:43.209063 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:43.209160 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:43.209673 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:00:43.210220 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:00:43.210322 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:43.210490 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 05:00:43.210548 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":598,"threads_started":1}
I20260925 05:00:43.210916 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:43.203923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:34935 (local address 127.0.0.1:58014)
0925 05:00:43.204159 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:43.204181 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:43.204337 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:43.205316 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:43.205322 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:43.205336 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:00:43.205524 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:43.205534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:43.206625 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:43.206632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:43.208279 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:43.208296 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:43.209208 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:43.209215 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:43.209420 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:43.209486 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:00:43.210325 (+ 839us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:00:43.210347 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 05:00:43.210436 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 05:00:43.210584 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 05:00:43.214946 525 raft_consensus.cc:1261] T 6cb95a496bf7481bb127f677bcd69e0a P 9b3474e03feb4e0f96eb9dc84b4f568c [term 1 FOLLOWER]: Refusing update from remote peer 7a2800457a6e43d9a8d54a9240e98c1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:00:43.215292 805 raft_consensus.cc:1261] T 6cb95a496bf7481bb127f677bcd69e0a P 8de2708cdd294f93ae6e26dcf829a870 [term 1 FOLLOWER]: Refusing update from remote peer 7a2800457a6e43d9a8d54a9240e98c1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:00:43.215641 917 consensus_queue.cc:1035] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:00:43.215956 877 consensus_queue.cc:1035] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:00:43.219094 875 tablet_replica.cc:439] T 6cb95a496bf7481bb127f677bcd69e0a P 9b3474e03feb4e0f96eb9dc84b4f568c: TxnStatusTablet state changed. Reason: New leader 7a2800457a6e43d9a8d54a9240e98c1e. Latest consensus state: current_term: 1 leader_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } } }
I20260925 05:00:43.219364 875 tablet_replica.cc:442] T 6cb95a496bf7481bb127f677bcd69e0a P 9b3474e03feb4e0f96eb9dc84b4f568c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:00:43.223738 919 mvcc.cc:205] Tried to move back new op lower bound from 7333119767401758720 to 7333119767114731520. Current Snapshot: MvccSnapshot[applied={T|T < 7333119767401758720}]
I20260925 05:00:43.224298 917 tablet_replica.cc:439] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } } }
I20260925 05:00:43.224556 917 tablet_replica.cc:442] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:00:43.231854 894 tablet_replica.cc:439] T 6cb95a496bf7481bb127f677bcd69e0a P 8de2708cdd294f93ae6e26dcf829a870: TxnStatusTablet state changed. Reason: New leader 7a2800457a6e43d9a8d54a9240e98c1e. Latest consensus state: current_term: 1 leader_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } } }
I20260925 05:00:43.232126 894 tablet_replica.cc:442] T 6cb95a496bf7481bb127f677bcd69e0a P 8de2708cdd294f93ae6e26dcf829a870: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:00:43.233340 877 tablet_replica.cc:439] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } } }
I20260925 05:00:43.233616 877 tablet_replica.cc:442] T 6cb95a496bf7481bb127f677bcd69e0a P 7a2800457a6e43d9a8d54a9240e98c1e: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:00:43.234110 915 mvcc.cc:205] Tried to move back new op lower bound from 7333119767401758720 to 7333119767114731520. Current Snapshot: MvccSnapshot[applied={T|T < 7333119767401758720}]
I20260925 05:00:43.241039 875 tablet_replica.cc:439] T 6cb95a496bf7481bb127f677bcd69e0a P 9b3474e03feb4e0f96eb9dc84b4f568c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } } }
I20260925 05:00:43.241307 875 tablet_replica.cc:442] T 6cb95a496bf7481bb127f677bcd69e0a P 9b3474e03feb4e0f96eb9dc84b4f568c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:00:43.242179 894 tablet_replica.cc:439] T 6cb95a496bf7481bb127f677bcd69e0a P 8de2708cdd294f93ae6e26dcf829a870: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3474e03feb4e0f96eb9dc84b4f568c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33069 } } peers { permanent_uuid: "7a2800457a6e43d9a8d54a9240e98c1e" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 34935 } } peers { permanent_uuid: "8de2708cdd294f93ae6e26dcf829a870" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34529 } } }
I20260925 05:00:43.242460 894 tablet_replica.cc:442] T 6cb95a496bf7481bb127f677bcd69e0a P 8de2708cdd294f93ae6e26dcf829a870: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:00:43.254901 924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:43.248289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:39251 (local address 127.30.247.2:34935)
0925 05:00:43.248453 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:43.248462 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:43.248892 (+ 430us) server_negotiation.cc:407] Connection header received
0925 05:00:43.249128 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:43.249134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:43.249207 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:43.249275 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:00:43.249845 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:43.250977 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:43.252900 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:43.253594 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:43.253699 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:43.254182 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:00:43.254316 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:00:43.254403 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:43.254550 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 05:00:43.254623 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 05:00:43.255303 932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:43.248184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:34935 (local address 127.30.247.3:39251)
0925 05:00:43.248805 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:43.248823 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:43.248990 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:43.249444 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:43.249449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:43.249460 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:00:43.249689 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:43.249698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:43.251146 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:43.251151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:43.252746 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:43.252759 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:43.253741 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:43.253748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:43.253934 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:43.254006 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:00:43.254422 (+ 416us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:00:43.254434 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 05:00:43.254517 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 05:00:43.254724 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":262,"threads_started":1}
I20260925 05:00:43.316602 937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:43.301375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:34529 (local address 127.30.247.2:33701)
0925 05:00:43.309473 (+ 8098us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:43.309492 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:43.309637 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:43.310244 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:43.310251 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:43.310263 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:00:43.310656 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:43.310678 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:43.312217 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:43.312223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:43.314172 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:43.314185 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:43.315202 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:43.315208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:43.315373 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:43.315446 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:00:43.315931 (+ 485us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:00:43.315948 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 05:00:43.316013 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 05:00:43.316140 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7175,"thread_start_us":7876,"threads_started":1}
I20260925 05:00:43.317274 938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:43.301442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:33701 (local address 127.30.247.3:34529)
0925 05:00:43.309228 (+ 7786us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:43.309235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:43.309757 (+ 522us) server_negotiation.cc:407] Connection header received
0925 05:00:43.309822 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:43.309828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:43.309910 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:43.309995 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:00:43.310866 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:43.312075 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:43.314344 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:43.314980 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:43.315564 (+ 584us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:43.315691 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:00:43.315816 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:00:43.315889 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:43.316684 (+ 795us) server_negotiation.cc:299] Negotiation successful
0925 05:00:43.316760 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7644,"thread_start_us":7523,"threads_started":1}
I20260925 05:00:43.353353 364 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:00:43.353600 364 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:00:43.356668 364 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e065cd42faf44fe08e2635e69533842a: Sending DeleteTablet for 3 replicas of tablet bcbf561275fe4e3788764bdc17708a3f
I20260925 05:00:43.357388 500 tablet_service.cc:1514] Processing DeleteTablet for tablet bcbf561275fe4e3788764bdc17708a3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:00:43 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:38258
I20260925 05:00:43.357544 639 tablet_service.cc:1514] Processing DeleteTablet for tablet bcbf561275fe4e3788764bdc17708a3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:00:43 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:58000
I20260925 05:00:43.357517 779 tablet_service.cc:1514] Processing DeleteTablet for tablet bcbf561275fe4e3788764bdc17708a3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:00:43 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:55034
I20260925 05:00:43.367326 942 tablet_replica.cc:330] T bcbf561275fe4e3788764bdc17708a3f P 7a2800457a6e43d9a8d54a9240e98c1e: stopping tablet replica
I20260925 05:00:43.367760 942 raft_consensus.cc:2229] T bcbf561275fe4e3788764bdc17708a3f P 7a2800457a6e43d9a8d54a9240e98c1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:00:43.368173 942 raft_consensus.cc:2258] T bcbf561275fe4e3788764bdc17708a3f P 7a2800457a6e43d9a8d54a9240e98c1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:00:43.370244 942 ts_tablet_manager.cc:1905] T bcbf561275fe4e3788764bdc17708a3f P 7a2800457a6e43d9a8d54a9240e98c1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:00:43.372589 941 tablet_replica.cc:330] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870: stopping tablet replica
I20260925 05:00:43.372789 941 raft_consensus.cc:2229] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:00:43.373131 941 raft_consensus.cc:2258] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:00:43.374166 941 ts_tablet_manager.cc:1905] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:00:43.374598 940 tablet_replica.cc:330] T bcbf561275fe4e3788764bdc17708a3f P 9b3474e03feb4e0f96eb9dc84b4f568c: stopping tablet replica
I20260925 05:00:43.378612 942 ts_tablet_manager.cc:1918] T bcbf561275fe4e3788764bdc17708a3f P 7a2800457a6e43d9a8d54a9240e98c1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:00:43.378904 940 raft_consensus.cc:2229] T bcbf561275fe4e3788764bdc17708a3f P 9b3474e03feb4e0f96eb9dc84b4f568c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:00:43.379253 940 raft_consensus.cc:2258] T bcbf561275fe4e3788764bdc17708a3f P 9b3474e03feb4e0f96eb9dc84b4f568c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:00:43.380328 940 ts_tablet_manager.cc:1905] T bcbf561275fe4e3788764bdc17708a3f P 9b3474e03feb4e0f96eb9dc84b4f568c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:00:43.381108 941 ts_tablet_manager.cc:1918] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:00:43.381250 941 log.cc:1196] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-2/wal/wals/bcbf561275fe4e3788764bdc17708a3f
I20260925 05:00:43.381728 941 ts_tablet_manager.cc:1939] T bcbf561275fe4e3788764bdc17708a3f P 8de2708cdd294f93ae6e26dcf829a870: Deleting consensus metadata
I20260925 05:00:43.382773 350 catalog_manager.cc:4841] TS 8de2708cdd294f93ae6e26dcf829a870 (127.30.247.3:34529): tablet bcbf561275fe4e3788764bdc17708a3f (table test-table [id=11ebce7973754b1ca017769e64e706fd]) successfully deleted
I20260925 05:00:43.379581 942 log.cc:1196] T bcbf561275fe4e3788764bdc17708a3f P 7a2800457a6e43d9a8d54a9240e98c1e: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-1/wal/wals/bcbf561275fe4e3788764bdc17708a3f
I20260925 05:00:43.383934 942 ts_tablet_manager.cc:1939] T bcbf561275fe4e3788764bdc17708a3f P 7a2800457a6e43d9a8d54a9240e98c1e: Deleting consensus metadata
I20260925 05:00:43.385007 350 catalog_manager.cc:4841] TS 7a2800457a6e43d9a8d54a9240e98c1e (127.30.247.2:34935): tablet bcbf561275fe4e3788764bdc17708a3f (table test-table [id=11ebce7973754b1ca017769e64e706fd]) successfully deleted
I20260925 05:00:43.387848 940 ts_tablet_manager.cc:1918] T bcbf561275fe4e3788764bdc17708a3f P 9b3474e03feb4e0f96eb9dc84b4f568c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:00:43.387992 940 log.cc:1196] T bcbf561275fe4e3788764bdc17708a3f P 9b3474e03feb4e0f96eb9dc84b4f568c: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312430481417-31708-0/minicluster-data/ts-0/wal/wals/bcbf561275fe4e3788764bdc17708a3f
I20260925 05:00:43.388499 940 ts_tablet_manager.cc:1939] T bcbf561275fe4e3788764bdc17708a3f P 9b3474e03feb4e0f96eb9dc84b4f568c: Deleting consensus metadata
I20260925 05:00:43.389458 351 catalog_manager.cc:4841] TS 9b3474e03feb4e0f96eb9dc84b4f568c (127.30.247.1:33069): tablet bcbf561275fe4e3788764bdc17708a3f (table test-table [id=11ebce7973754b1ca017769e64e706fd]) successfully deleted
Sep 25 05:00:43 dist-test-slave-5n86 krb5kdc[320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312441, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.247.1@KRBTEST.COM
I20260925 05:00:43.404263 944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:43.375516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38274 (local address 127.30.247.1:33069)
0925 05:00:43.388444 (+ 12928us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:43.388452 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:43.388475 (+ 23us) server_negotiation.cc:407] Connection header received
0925 05:00:43.388576 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:43.388583 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:43.388675 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:43.388807 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:43.390693 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:43.391388 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:43.393197 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:43.393616 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:43.393717 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:43.398447 (+ 4730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:43.398460 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:43.398473 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:43.398506 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:43.401552 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:43.402212 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:43.402219 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:43.402222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:43.402347 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:43.402739 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:43.402743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:43.402745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:43.403089 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:43.403377 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:43.403691 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 05:00:43.403934 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12778,"thread_start_us":12675,"threads_started":1}
W20260925 05:00:43.405067 566 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38274
I20260925 05:00:43.421944 947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:43.407857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45076 (local address 127.30.247.62:45447)
0925 05:00:43.408395 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:43.408403 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:43.408855 (+ 452us) server_negotiation.cc:407] Connection header received
0925 05:00:43.409073 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:43.409091 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:43.409169 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:43.409256 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:43.410659 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:43.411375 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:43.413109 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:43.413528 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:43.413643 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:43.416300 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:43.416308 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:43.416312 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:43.416360 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:43.419280 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:43.420021 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:43.420025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:43.420027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:43.420147 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:43.420502 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:43.420504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:43.420506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:43.420851 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:43.421045 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:43.421404 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 05:00:43.421570 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":231,"threads_started":1}
W20260925 05:00:43.422526 364 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45076
I20260925 05:00:43.424458 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 436
I20260925 05:00:43.439934 944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:43.425299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38288 (local address 127.30.247.1:33069)
0925 05:00:43.425663 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:43.425667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:43.428985 (+ 3318us) server_negotiation.cc:407] Connection header received
0925 05:00:43.429214 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:43.429218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:43.429277 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:43.429382 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:43.430427 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:43.431105 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:43.432248 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:43.432732 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:43.432861 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:43.434983 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:43.434988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:43.434992 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:43.435021 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:43.437498 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:43.438113 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:43.438120 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:43.438128 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:43.438239 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:43.438590 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:43.438596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:43.438599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:43.438845 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:43.439091 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:43.439384 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 05:00:43.439604 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245}
W20260925 05:00:43.440487 566 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38288
W20260925 05:00:43.441107 31708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 436: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:00:43.441282 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 436
I20260925 05:00:43.463629 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 575
Sep 25 05:00:43 dist-test-slave-5n86 krb5kdc[320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312441, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.247.2@KRBTEST.COM
I20260925 05:00:43.480078 924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:43.464376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58016 (local address 127.30.247.2:34935)
0925 05:00:43.464610 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:43.464615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:43.464636 (+ 21us) server_negotiation.cc:407] Connection header received
0925 05:00:43.464743 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:43.464749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:43.464826 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:43.464934 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:43.466085 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:43.467293 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:43.468576 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:43.469182 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:43.469309 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:43.473129 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:43.473140 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:43.473157 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:43.473197 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:43.477352 (+ 4155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:43.477972 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:43.477981 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:43.477985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:43.478135 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:43.478477 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:43.478484 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:43.478493 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:43.478925 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:43.479242 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:43.479441 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 05:00:43.479675 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 05:00:43.480904 705 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58016
W20260925 05:00:43.481549 31708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 575: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:00:43.481761 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 575
I20260925 05:00:43.500944 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 714
Sep 25 05:00:43 dist-test-slave-5n86 krb5kdc[320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312441, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.247.3@KRBTEST.COM
I20260925 05:00:43.514940 938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:43.501724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55054 (local address 127.30.247.3:34529)
0925 05:00:43.501890 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:43.501894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:43.501931 (+ 37us) server_negotiation.cc:407] Connection header received
0925 05:00:43.501981 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:43.501984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:43.502049 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:43.502131 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:43.503276 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:43.504001 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:43.505356 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:43.505756 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:43.505857 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:43.509978 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:43.509991 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:43.510006 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:43.510035 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:43.512768 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:43.513330 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:43.513336 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:43.513339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:43.513458 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:43.513762 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:43.513766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:43.513768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:43.514058 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:43.514278 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:43.514461 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 05:00:43.514631 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 05:00:43.515587 845 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55054
W20260925 05:00:43.516259 31708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 714: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:00:43.516397 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 714
I20260925 05:00:43.537557 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 333
I20260925 05:00:43.538661 405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:00:43.651298 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 333
2026-09-25T05:00:43Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5086 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:00:43 dist-test-slave-5n86 krb5kdc[953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:00:43 dist-test-slave-5n86 krb5kdc[953](info): set up 2 sockets
Sep 25 05:00:43 dist-test-slave-5n86 krb5kdc[953](info): commencing operation
krb5kdc: starting...
W20260925 05:00:44.728998 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:00:44 dist-test-slave-5n86 krb5kdc[953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:00:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:00:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:00:44.905328 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:35363
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:35363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:45.211637 966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:45.211953 966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:45.212116 966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:45.219182 966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:00:45.219313 966 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 05:00:45.219412 966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:45.219537 966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:45.219623 966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:00:45.219712 966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:00:45.229825 966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:35363
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:35363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:45.231007 966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:45.232756 966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:45.244527 972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:45.244632 974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:45.244859 971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:45.245252 966 server_base.cc:1034] running on GCE node
I20260925 05:00:45.245982 966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:45.247258 966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:45.248513 966 hybrid_clock.cc:648] HybridClock initialized: now 1790312445248452 us; error 49 us; skew 500 ppm
I20260925 05:00:45.248903 966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:45 dist-test-slave-5n86 krb5kdc[953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312445, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:45.251904 966 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:00:45.253397 966 webserver.cc:460] Webserver started at http://127.30.247.62:32851/ using document root <none> and password file <none>
I20260925 05:00:45.254076 966 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:45.254204 966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:45.254501 966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:45.257364 966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "8e75816c69a345df9807f8da66ce85dd"
format_stamp: "Formatted at 2026-09-25 05:00:45 on dist-test-slave-5n86"
server_key: "6e0795e2de9b6d7de95d44b1903c77fb"
server_key_iv: "50bb4264b0827c427fe8395fb98091e3"
server_key_version: "encryptionkey@0"
I20260925 05:00:45.258100 966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "8e75816c69a345df9807f8da66ce85dd"
format_stamp: "Formatted at 2026-09-25 05:00:45 on dist-test-slave-5n86"
server_key: "6e0795e2de9b6d7de95d44b1903c77fb"
server_key_iv: "50bb4264b0827c427fe8395fb98091e3"
server_key_version: "encryptionkey@0"
I20260925 05:00:45.262889 966 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 05:00:45.266206 981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:45.267458 966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:00:45.267653 966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "8e75816c69a345df9807f8da66ce85dd"
format_stamp: "Formatted at 2026-09-25 05:00:45 on dist-test-slave-5n86"
server_key: "6e0795e2de9b6d7de95d44b1903c77fb"
server_key_iv: "50bb4264b0827c427fe8395fb98091e3"
server_key_version: "encryptionkey@0"
I20260925 05:00:45.267858 966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:45.292066 966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:45.296137 966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:45.296417 966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:45.317659 966 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:35363
I20260925 05:00:45.317677 1043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:35363 every 8 connection(s)
I20260925 05:00:45.318924 966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:00:45.320266 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 966
I20260925 05:00:45.320475 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:00:45.321295 31708 external_mini_cluster.cc:1428] Setting key 442dbfc8f4b14757c3776e9bba165dd1
I20260925 05:00:45.323669 1044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 05:00:45 dist-test-slave-5n86 krb5kdc[953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:00:45.336390 1044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e75816c69a345df9807f8da66ce85dd: Bootstrap starting.
I20260925 05:00:45.340457 1044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e75816c69a345df9807f8da66ce85dd: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:45.341662 1044 log.cc:824] T 00000000000000000000000000000000 P 8e75816c69a345df9807f8da66ce85dd: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:45.344944 1047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:45.323291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38168 (local address 127.30.247.62:35363)
0925 05:00:45.324054 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:45.324071 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:45.324110 (+ 39us) server_negotiation.cc:407] Connection header received
0925 05:00:45.325659 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:45.325680 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:45.326039 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:45.326372 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:45.328023 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:45.329436 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:45.331165 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:45.331797 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:45.331921 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:45.336582 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:45.336594 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:45.336611 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:45.336652 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:45.340461 (+ 3809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:45.341165 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:45.341173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:45.341182 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:45.341386 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:45.341770 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:45.341774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:45.341777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:45.342454 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:45.342676 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:45.343259 (+ 583us) server_negotiation.cc:299] Negotiation successful
0925 05:00:45.343652 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":383,"threads_started":1}
I20260925 05:00:45.344960 1044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e75816c69a345df9807f8da66ce85dd: No bootstrap required, opened a new log
I20260925 05:00:45.352382 1044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e75816c69a345df9807f8da66ce85dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e75816c69a345df9807f8da66ce85dd" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35363 } }
I20260925 05:00:45.352797 1044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e75816c69a345df9807f8da66ce85dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:45.352872 1044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e75816c69a345df9807f8da66ce85dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e75816c69a345df9807f8da66ce85dd, State: Initialized, Role: FOLLOWER
I20260925 05:00:45.353500 1044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e75816c69a345df9807f8da66ce85dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e75816c69a345df9807f8da66ce85dd" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35363 } }
I20260925 05:00:45.353746 1044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e75816c69a345df9807f8da66ce85dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:00:45.353930 1044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e75816c69a345df9807f8da66ce85dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:00:45.354137 1044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e75816c69a345df9807f8da66ce85dd [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:45.356177 1044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e75816c69a345df9807f8da66ce85dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e75816c69a345df9807f8da66ce85dd" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35363 } }
I20260925 05:00:45.356755 1044 leader_election.cc:302] T 00000000000000000000000000000000 P 8e75816c69a345df9807f8da66ce85dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e75816c69a345df9807f8da66ce85dd; no voters:
I20260925 05:00:45.357345 1044 leader_election.cc:288] T 00000000000000000000000000000000 P 8e75816c69a345df9807f8da66ce85dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:00:45.357481 1049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e75816c69a345df9807f8da66ce85dd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:00:45.358245 1049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e75816c69a345df9807f8da66ce85dd [term 1 LEADER]: Becoming Leader. State: Replica: 8e75816c69a345df9807f8da66ce85dd, State: Running, Role: LEADER
I20260925 05:00:45.358877 1049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e75816c69a345df9807f8da66ce85dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e75816c69a345df9807f8da66ce85dd" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35363 } }
I20260925 05:00:45.359546 1044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e75816c69a345df9807f8da66ce85dd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:00:45.362088 1051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e75816c69a345df9807f8da66ce85dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e75816c69a345df9807f8da66ce85dd. Latest consensus state: current_term: 1 leader_uuid: "8e75816c69a345df9807f8da66ce85dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e75816c69a345df9807f8da66ce85dd" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35363 } } }
I20260925 05:00:45.362229 1050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e75816c69a345df9807f8da66ce85dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e75816c69a345df9807f8da66ce85dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e75816c69a345df9807f8da66ce85dd" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35363 } } }
I20260925 05:00:45.362556 1051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e75816c69a345df9807f8da66ce85dd [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:45.362632 1050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e75816c69a345df9807f8da66ce85dd [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:45.366585 1058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:00:45.370777 1058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:00:45.378643 1058 catalog_manager.cc:1347] Generated new cluster ID: 052d0dbf6b9342db9c4af8bfc6968855
I20260925 05:00:45.378767 1058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:00:45.407382 1058 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:00:45.408820 1058 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:00:45.417343 1058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e75816c69a345df9807f8da66ce85dd: Generated new TSK 0
I20260925 05:00:45.418155 1058 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
I20260925 05:00:45.464049 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:0
--local_ip_for_outbound_sockets=127.30.247.1
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:35363
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:45.781239 1070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:45.781502 1070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:45.781674 1070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:45.788762 1070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:45.788877 1070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:45.789034 1070 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 05:00:45.789120 1070 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:00:45.789249 1070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:00:45.798563 1070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.247.62:35363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:45.799888 1070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:45.801221 1070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:45.814079 1077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:45.814107 1076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:45.814034 1079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:45.814723 1070 server_base.cc:1034] running on GCE node
I20260925 05:00:45.815349 1070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:45.816174 1070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:45.817387 1070 hybrid_clock.cc:648] HybridClock initialized: now 1790312445817350 us; error 54 us; skew 500 ppm
I20260925 05:00:45.817694 1070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:45 dist-test-slave-5n86 krb5kdc[953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312445, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:45.820762 1070 init.cc:377] Logged in from keytab as kudu/127.30.247.1@KRBTEST.COM (short username kudu)
I20260925 05:00:45.821990 1070 webserver.cc:460] Webserver started at http://127.30.247.1:36547/ using document root <none> and password file <none>
I20260925 05:00:45.822562 1070 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:45.822664 1070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:45.822896 1070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:45.825557 1070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-0/data/instance:
uuid: "57227d8f465c4c4783262e70f0929206"
format_stamp: "Formatted at 2026-09-25 05:00:45 on dist-test-slave-5n86"
server_key: "b1d4009cf8d95d8e163322fb262229bb"
server_key_iv: "a4638de89072d89d14001e685743c4be"
server_key_version: "encryptionkey@0"
I20260925 05:00:45.826241 1070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance:
uuid: "57227d8f465c4c4783262e70f0929206"
format_stamp: "Formatted at 2026-09-25 05:00:45 on dist-test-slave-5n86"
server_key: "b1d4009cf8d95d8e163322fb262229bb"
server_key_iv: "a4638de89072d89d14001e685743c4be"
server_key_version: "encryptionkey@0"
I20260925 05:00:45.830853 1070 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 05:00:45.833981 1086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:45.835197 1070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 05:00:45.835325 1070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "57227d8f465c4c4783262e70f0929206"
format_stamp: "Formatted at 2026-09-25 05:00:45 on dist-test-slave-5n86"
server_key: "b1d4009cf8d95d8e163322fb262229bb"
server_key_iv: "a4638de89072d89d14001e685743c4be"
server_key_version: "encryptionkey@0"
I20260925 05:00:45.835467 1070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:45.852996 1070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:45.857271 1070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:45.857498 1070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:45.859978 1070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:00:45.860143 1070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:45.860272 1070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:00:45.860379 1070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 05:00:45 dist-test-slave-5n86 krb5kdc[953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312445, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:00:45.885322 1095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:45.860663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:52091 (local address 127.30.247.62:35363)
0925 05:00:45.863018 (+ 2355us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:45.863024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:45.863046 (+ 22us) server_negotiation.cc:407] Connection header received
0925 05:00:45.865085 (+ 2039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:45.865091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:45.865189 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:45.865311 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:45.867576 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:45.868718 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:45.872066 (+ 3348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:45.872677 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:45.872815 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:45.877859 (+ 5044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:45.877863 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:45.877867 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:45.877899 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:45.880734 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:45.881869 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:45.881878 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:45.881883 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:45.882056 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:45.882638 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:45.882642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:45.882644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:45.882914 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:45.883025 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:45.884751 (+ 1726us) server_negotiation.cc:299] Negotiation successful
0925 05:00:45.884974 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2203,"thread_start_us":2096,"threads_started":1}
I20260925 05:00:45.885953 1096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:45.861313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:35363 (local address 127.30.247.1:52091)
0925 05:00:45.862177 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:45.862236 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:45.864810 (+ 2574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:45.865549 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:45.865557 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:45.866317 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:45.867384 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:45.867412 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:45.868898 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:45.868905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:45.871886 (+ 2981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:45.871907 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:45.872914 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:45.872921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:45.873146 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:45.873968 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:45.873991 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:45.877603 (+ 3612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:45.880910 (+ 3307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:45.880923 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:45.880951 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:45.881722 (+ 771us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:45.882199 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:45.882204 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:45.882209 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:45.882512 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:45.883031 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:45.883037 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:45.883562 (+ 525us) client_negotiation.cc:769] Sending connection context
0925 05:00:45.883950 (+ 388us) client_negotiation.cc:240] Negotiation successful
0925 05:00:45.884376 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":356,"threads_started":1}
I20260925 05:00:45.910406 1070 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:44329
I20260925 05:00:45.910501 1207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:44329 every 8 connection(s)
I20260925 05:00:45.910621 1163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:45.893608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:35363 (local address 127.30.247.1:58861)
0925 05:00:45.894327 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:45.894341 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:45.894488 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:45.894988 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:45.894994 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:45.895751 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:45.896246 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:45.896264 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:45.897904 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:45.897909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:45.899961 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:45.899982 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:45.900925 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:45.900931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:45.901126 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:45.901869 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:45.901902 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:45.903191 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:45.907195 (+ 4004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:45.907201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:45.907206 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:45.907926 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:45.908393 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:45.908400 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:45.908404 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:45.908620 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:45.909136 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:45.909143 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:45.909401 (+ 258us) client_negotiation.cc:769] Sending connection context
0925 05:00:45.909586 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 05:00:45.909966 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":411,"threads_started":1}
I20260925 05:00:45.911481 1095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:45.893111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:58861 (local address 127.30.247.62:35363)
0925 05:00:45.893275 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:45.893279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:45.894421 (+ 1142us) server_negotiation.cc:407] Connection header received
0925 05:00:45.894606 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:45.894614 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:45.894702 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:45.894803 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:45.896455 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:45.897696 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:45.900186 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:45.900738 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:45.900881 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:45.903386 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:45.903391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:45.903394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:45.903422 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:45.907034 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:45.908065 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:45.908074 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:45.908080 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:45.908242 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:45.908736 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:45.908739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:45.908741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:45.908998 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:45.909104 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:45.910948 (+ 1844us) server_negotiation.cc:299] Negotiation successful
0925 05:00:45.911167 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 05:00:45.911957 1070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:00:45.914667 1208 heartbeater.cc:347] Connected to a master server at 127.30.247.62:35363
I20260925 05:00:45.914899 1208 heartbeater.cc:464] Registering TS with master...
I20260925 05:00:45.915637 1208 heartbeater.cc:511] Master 127.30.247.62:35363 requested a full tablet report, sending...
I20260925 05:00:45.917606 998 ts_manager.cc:194] Registered new tserver with Master: 57227d8f465c4c4783262e70f0929206 (127.30.247.1:44329)
I20260925 05:00:45.918955 998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.1@KRBTEST.COM'} at 127.30.247.1:52091
I20260925 05:00:45.920922 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 1070
I20260925 05:00:45.921087 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance
I20260925 05:00:45.921629 31708 external_mini_cluster.cc:1428] Setting key 9bfe2ab6d2f377a43c1908d10c080391
WARNING: no policy specified for kudu/127.30.247.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
I20260925 05:00:45.960481 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:0
--local_ip_for_outbound_sockets=127.30.247.2
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:35363
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:46.271554 1212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:46.271874 1212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:46.272058 1212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:46.279181 1212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:46.279278 1212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:46.279359 1212 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 05:00:46.279412 1212 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:00:46.279472 1212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:00:46.288910 1212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.247.62:35363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:46.290140 1212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:46.291368 1212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:46.303869 1217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:46.303874 1218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:46.303856 1220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:46.305336 1212 server_base.cc:1034] running on GCE node
I20260925 05:00:46.305869 1212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:46.306696 1212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:46.307906 1212 hybrid_clock.cc:648] HybridClock initialized: now 1790312446307861 us; error 43 us; skew 500 ppm
I20260925 05:00:46.308220 1212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:46 dist-test-slave-5n86 krb5kdc[953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312446, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:46.317020 1212 init.cc:377] Logged in from keytab as kudu/127.30.247.2@KRBTEST.COM (short username kudu)
I20260925 05:00:46.318557 1212 webserver.cc:460] Webserver started at http://127.30.247.2:33401/ using document root <none> and password file <none>
I20260925 05:00:46.319228 1212 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:46.319371 1212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:46.319674 1212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:46.322474 1212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-1/data/instance:
uuid: "105aa6bd36964ed6b923cc2e4a9e31c9"
format_stamp: "Formatted at 2026-09-25 05:00:46 on dist-test-slave-5n86"
server_key: "60e24210d64f8d43accd02d5d23a473d"
server_key_iv: "01e234849305170628fb5463f9a51902"
server_key_version: "encryptionkey@0"
I20260925 05:00:46.323215 1212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance:
uuid: "105aa6bd36964ed6b923cc2e4a9e31c9"
format_stamp: "Formatted at 2026-09-25 05:00:46 on dist-test-slave-5n86"
server_key: "60e24210d64f8d43accd02d5d23a473d"
server_key_iv: "01e234849305170628fb5463f9a51902"
server_key_version: "encryptionkey@0"
I20260925 05:00:46.328300 1212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 05:00:46.331624 1227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:46.332888 1212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:00:46.333055 1212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "105aa6bd36964ed6b923cc2e4a9e31c9"
format_stamp: "Formatted at 2026-09-25 05:00:46 on dist-test-slave-5n86"
server_key: "60e24210d64f8d43accd02d5d23a473d"
server_key_iv: "01e234849305170628fb5463f9a51902"
server_key_version: "encryptionkey@0"
I20260925 05:00:46.333247 1212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:46.354049 1212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:46.358517 1212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:46.358816 1212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:46.361627 1212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:00:46.361771 1212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:46.361907 1212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:00:46.361999 1212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 05:00:46 dist-test-slave-5n86 krb5kdc[953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312446, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:00:46.384922 1095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:46.362294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:44585 (local address 127.30.247.62:35363)
0925 05:00:46.362467 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:46.362472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:46.364030 (+ 1558us) server_negotiation.cc:407] Connection header received
0925 05:00:46.367363 (+ 3333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:46.367401 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:46.367582 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:46.367709 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:46.370313 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:46.371334 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:46.373083 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:46.373482 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:46.373613 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:46.377451 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:46.377457 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:46.377462 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:46.377505 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:46.380856 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:46.382042 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:46.382050 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:46.382055 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:46.382226 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:46.382842 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:46.382847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:46.382851 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:46.383199 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:46.383366 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:46.384330 (+ 964us) server_negotiation.cc:299] Negotiation successful
0925 05:00:46.384568 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 05:00:46.385943 1236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:46.362916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:35363 (local address 127.30.247.2:44585)
0925 05:00:46.363830 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:46.363889 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:46.366929 (+ 3040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:46.368006 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:46.368026 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:46.368978 (+ 952us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:46.370091 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:46.370123 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:46.371512 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:46.371516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:46.372942 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:46.372953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:46.373725 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:46.373735 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:46.373945 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:46.374657 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:46.374682 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:46.377192 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:46.381025 (+ 3833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:46.381037 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:46.381059 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:46.381893 (+ 834us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:46.382349 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:46.382354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:46.382358 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:46.382663 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:46.383443 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:46.383450 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:46.383910 (+ 460us) client_negotiation.cc:769] Sending connection context
0925 05:00:46.384240 (+ 330us) client_negotiation.cc:240] Negotiation successful
0925 05:00:46.384793 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":355,"threads_started":1}
I20260925 05:00:46.406244 1095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:46.392659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:35687 (local address 127.30.247.62:35363)
0925 05:00:46.392857 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:46.392863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:46.393512 (+ 649us) server_negotiation.cc:407] Connection header received
0925 05:00:46.393699 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:46.393704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:46.393771 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:46.393873 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:46.394723 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:46.395439 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:46.396834 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:46.397386 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:46.397522 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:46.400083 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:46.400088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:46.400091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:46.400118 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:46.403281 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:46.404196 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:46.404203 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:46.404210 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:46.404345 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:46.404813 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:46.404818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:46.404822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:46.405153 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:46.405303 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:46.405683 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 05:00:46.405895 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 05:00:46.406366 1281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:46.392661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:35363 (local address 127.30.247.2:35687)
0925 05:00:46.393393 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:46.393412 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:46.393573 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:46.393895 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:46.393900 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:46.394320 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:46.394584 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:46.394592 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:46.395574 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:46.395582 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:46.396682 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:46.396693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:46.397620 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:46.397633 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:46.397920 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:46.398580 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:46.398610 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:46.399888 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:46.403420 (+ 3532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:46.403429 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:46.403432 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:46.404049 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:46.404453 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:46.404459 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:46.404462 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:46.404663 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:46.405273 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:46.405278 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:46.405454 (+ 176us) client_negotiation.cc:769] Sending connection context
0925 05:00:46.405585 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 05:00:46.405896 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":450,"threads_started":1}
I20260925 05:00:46.425087 1212 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:46397
I20260925 05:00:46.425129 1346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:46397 every 8 connection(s)
I20260925 05:00:46.426299 1212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:00:46.427695 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 1212
I20260925 05:00:46.427878 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance
I20260925 05:00:46.428334 1347 heartbeater.cc:347] Connected to a master server at 127.30.247.62:35363
I20260925 05:00:46.428417 31708 external_mini_cluster.cc:1428] Setting key 4ac8683afc65a76986e728fff8106d17
I20260925 05:00:46.428594 1347 heartbeater.cc:464] Registering TS with master...
I20260925 05:00:46.429211 1347 heartbeater.cc:511] Master 127.30.247.62:35363 requested a full tablet report, sending...
I20260925 05:00:46.430626 998 ts_manager.cc:194] Registered new tserver with Master: 105aa6bd36964ed6b923cc2e4a9e31c9 (127.30.247.2:46397)
I20260925 05:00:46.431833 998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.2@KRBTEST.COM'} at 127.30.247.2:44585
WARNING: no policy specified for kudu/127.30.247.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
I20260925 05:00:46.468773 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:0
--local_ip_for_outbound_sockets=127.30.247.3
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:35363
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:46.778739 1351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:46.779127 1351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:46.779330 1351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:46.787016 1351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:46.787109 1351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:46.787267 1351 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 05:00:46.787344 1351 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:00:46.787446 1351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:00:46.797683 1351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.247.62:35363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:46.798823 1351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:46.800035 1351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:46.812897 1360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:46.812829 1358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:46.812932 1357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:46.813513 1351 server_base.cc:1034] running on GCE node
I20260925 05:00:46.814098 1351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:46.814911 1351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:46.816124 1351 hybrid_clock.cc:648] HybridClock initialized: now 1790312446816083 us; error 53 us; skew 500 ppm
I20260925 05:00:46.816447 1351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:46 dist-test-slave-5n86 krb5kdc[953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312446, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:46.819571 1351 init.cc:377] Logged in from keytab as kudu/127.30.247.3@KRBTEST.COM (short username kudu)
I20260925 05:00:46.820936 1351 webserver.cc:460] Webserver started at http://127.30.247.3:35713/ using document root <none> and password file <none>
I20260925 05:00:46.821564 1351 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:46.821704 1351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:46.822000 1351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:46.824807 1351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-2/data/instance:
uuid: "8463fe556af94c4ea931e4d060f9d7a5"
format_stamp: "Formatted at 2026-09-25 05:00:46 on dist-test-slave-5n86"
server_key: "ccd32380b26b1385c399f8f32be33a24"
server_key_iv: "f8cacf9a836b7d65f6ad2b3658751d7f"
server_key_version: "encryptionkey@0"
I20260925 05:00:46.825536 1351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance:
uuid: "8463fe556af94c4ea931e4d060f9d7a5"
format_stamp: "Formatted at 2026-09-25 05:00:46 on dist-test-slave-5n86"
server_key: "ccd32380b26b1385c399f8f32be33a24"
server_key_iv: "f8cacf9a836b7d65f6ad2b3658751d7f"
server_key_version: "encryptionkey@0"
I20260925 05:00:46.830327 1351 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 05:00:46.833612 1367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:46.834882 1351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:00:46.835065 1351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "8463fe556af94c4ea931e4d060f9d7a5"
format_stamp: "Formatted at 2026-09-25 05:00:46 on dist-test-slave-5n86"
server_key: "ccd32380b26b1385c399f8f32be33a24"
server_key_iv: "f8cacf9a836b7d65f6ad2b3658751d7f"
server_key_version: "encryptionkey@0"
I20260925 05:00:46.835265 1351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:46.865087 1351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:46.869510 1351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:46.869814 1351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:46.872458 1351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:00:46.872596 1351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:46.872745 1351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:00:46.872861 1351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 05:00:46 dist-test-slave-5n86 krb5kdc[953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312446, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:00:46.896170 1095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:46.873428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:35137 (local address 127.30.247.62:35363)
0925 05:00:46.873604 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:46.873610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:46.875528 (+ 1918us) server_negotiation.cc:407] Connection header received
0925 05:00:46.877782 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:46.877789 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:46.877880 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:46.877986 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:46.880389 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:46.881189 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:46.882749 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:46.883152 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:46.883286 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:46.888254 (+ 4968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:46.888262 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:46.888268 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:46.888314 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:46.891927 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:46.892931 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:46.892939 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:46.892943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:46.893113 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:46.893835 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:46.893843 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:46.893848 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:46.894306 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:46.894466 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:46.895290 (+ 824us) server_negotiation.cc:299] Negotiation successful
0925 05:00:46.895622 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 05:00:46.896724 1376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:46.874323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:35363 (local address 127.30.247.3:35137)
0925 05:00:46.875317 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:46.875373 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:46.877434 (+ 2061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:46.878518 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:46.878534 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:46.879342 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:46.880205 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:46.880224 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:46.881340 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:46.881345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:46.882610 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:46.882621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:46.883379 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:46.883385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:46.883580 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:46.884302 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:46.884344 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:46.887933 (+ 3589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:46.892061 (+ 4128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:46.892076 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:46.892096 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:46.892798 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:46.893247 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:46.893260 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:46.893266 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:46.893626 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:46.894461 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:46.894470 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:46.894812 (+ 342us) client_negotiation.cc:769] Sending connection context
0925 05:00:46.895140 (+ 328us) client_negotiation.cc:240] Negotiation successful
0925 05:00:46.895544 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":489,"threads_started":1}
I20260925 05:00:46.919708 1095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:46.903256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:52871 (local address 127.30.247.62:35363)
0925 05:00:46.903438 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:46.903443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:46.903627 (+ 184us) server_negotiation.cc:407] Connection header received
0925 05:00:46.903829 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:46.903834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:46.903903 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:46.904001 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:46.905171 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:46.906411 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:46.907737 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:46.908685 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:46.908830 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:46.913677 (+ 4847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:46.913681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:46.913685 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:46.913712 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:46.916230 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:46.917364 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:46.917370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:46.917373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:46.917508 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:46.917970 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:46.917976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:46.917981 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:46.918357 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:46.918508 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:46.919118 (+ 610us) server_negotiation.cc:299] Negotiation successful
0925 05:00:46.919336 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 05:00:46.919835 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:46.902779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:35363 (local address 127.30.247.3:52871)
0925 05:00:46.903528 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:46.903543 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:46.903692 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:46.904023 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:46.904027 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:46.904505 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:46.904986 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:46.905003 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:46.906561 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:46.906567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:46.907588 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:46.907599 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:46.908906 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:46.908913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:46.909051 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:46.912743 (+ 3692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:46.912760 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:46.913469 (+ 709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:46.916421 (+ 2952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:46.916430 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:46.916436 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:46.917250 (+ 814us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:46.917625 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:46.917633 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:46.917640 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:46.917817 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:46.918588 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:46.918596 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:46.918842 (+ 246us) client_negotiation.cc:769] Sending connection context
0925 05:00:46.919007 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 05:00:46.919346 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":462,"threads_started":1}
I20260925 05:00:46.921665 1208 heartbeater.cc:503] Master 127.30.247.62:35363 was elected leader, sending a full tablet report...
I20260925 05:00:46.932348 1351 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:45275
I20260925 05:00:46.932384 1487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:45275 every 8 connection(s)
I20260925 05:00:46.933532 1351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:00:46.934979 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 1351
I20260925 05:00:46.935156 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance
I20260925 05:00:46.935472 1488 heartbeater.cc:347] Connected to a master server at 127.30.247.62:35363
I20260925 05:00:46.935685 1488 heartbeater.cc:464] Registering TS with master...
I20260925 05:00:46.935729 31708 external_mini_cluster.cc:1428] Setting key e6f909aa984139afe9b3d2d901c9100e
I20260925 05:00:46.936447 1488 heartbeater.cc:511] Master 127.30.247.62:35363 requested a full tablet report, sending...
I20260925 05:00:46.938289 998 ts_manager.cc:194] Registered new tserver with Master: 8463fe556af94c4ea931e4d060f9d7a5 (127.30.247.3:45275)
I20260925 05:00:46.939303 998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.3@KRBTEST.COM'} at 127.30.247.3:35137
I20260925 05:00:46.939401 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 05:00:46 dist-test-slave-5n86 krb5kdc[953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 05:00:46 dist-test-slave-5n86 krb5kdc[953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:00:46.984587 1095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:46.970710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38180 (local address 127.30.247.62:35363)
0925 05:00:46.970965 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:46.970975 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:46.971081 (+ 106us) server_negotiation.cc:407] Connection header received
0925 05:00:46.971290 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:46.971294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:46.971356 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:46.971439 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:46.972967 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:46.973690 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:46.975107 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:46.975516 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:46.975617 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:46.979398 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:46.979404 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:46.979407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:46.979439 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:46.981990 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:46.982621 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:46.982629 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:46.982635 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:46.982757 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:46.983125 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:46.983130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:46.983133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:46.983399 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:46.983653 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:46.983959 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 05:00:46.984189 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260925 05:00:46.988745 998 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:00:46.995373 998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:00:47.021634 1502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:47.007428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:44329 (local address 127.0.0.1:57844)
0925 05:00:47.008225 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:47.008285 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:47.008449 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:47.015645 (+ 7196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:47.015649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:47.015674 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:00:47.016006 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:47.016014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.017772 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.017779 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:47.019335 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:47.019347 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.020909 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.020916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:47.021139 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:47.021180 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 05:00:47.021262 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 05:00:47.021388 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":5831,"threads_started":1}
I20260925 05:00:47.022701 1503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:47.007672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57844 (local address 127.30.247.1:44329)
0925 05:00:47.015187 (+ 7515us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:47.015197 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:47.015225 (+ 28us) server_negotiation.cc:407] Connection header received
0925 05:00:47.015324 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:47.015329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:47.015474 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:47.015649 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:00:47.016162 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.017620 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.019484 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.020682 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.022094 (+ 1412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:47.022232 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:47.022285 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 05:00:47.022396 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7357,"thread_start_us":7202,"threads_started":1}
I20260925 05:00:47.022946 1501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:47.007479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:45275 (local address 127.0.0.1:38176)
0925 05:00:47.013632 (+ 6153us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:47.013656 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:47.013810 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:47.014678 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:47.014686 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:47.014715 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:00:47.015097 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:47.015112 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.017303 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.017308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:47.020088 (+ 2780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:47.020102 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.022118 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.022124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:47.022326 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:47.022358 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 05:00:47.022430 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 05:00:47.022572 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5964,"mutex_wait_us":197,"thread_start_us":438,"threads_started":1}
I20260925 05:00:47.023308 1505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:47.008177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38176 (local address 127.30.247.3:45275)
0925 05:00:47.009025 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:47.009031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:47.013945 (+ 4914us) server_negotiation.cc:407] Connection header received
0925 05:00:47.014049 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:47.014058 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:47.014329 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:47.014522 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:00:47.015329 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.017117 (+ 1788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.020282 (+ 3165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.021628 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.021729 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:47.021953 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:47.022497 (+ 544us) server_negotiation.cc:299] Negotiation successful
0925 05:00:47.022644 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":497,"threads_started":1}
I20260925 05:00:47.023612 1504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:47.007827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:46397 (local address 127.0.0.1:48826)
0925 05:00:47.013994 (+ 6167us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:47.014017 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:47.014174 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:47.015150 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:47.015154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:47.015172 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:00:47.015579 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:47.015592 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.017311 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.017314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:47.020092 (+ 2778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:47.020102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.022076 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.022084 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:47.022259 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:47.022343 (+ 84us) client_negotiation.cc:769] Sending connection context
0925 05:00:47.022430 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 05:00:47.023292 (+ 862us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6012,"mutex_wait_us":951,"thread_start_us":5854,"threads_started":1}
I20260925 05:00:47.024041 1506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:47.008408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48826 (local address 127.30.247.2:46397)
0925 05:00:47.013065 (+ 4657us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:47.013078 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:47.014332 (+ 1254us) server_negotiation.cc:407] Connection header received
0925 05:00:47.014458 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:47.014469 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:47.014754 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:47.014951 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:00:47.015787 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.017181 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.020264 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.021621 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.021762 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:47.021923 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:47.022542 (+ 619us) server_negotiation.cc:299] Negotiation successful
0925 05:00:47.022721 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4451,"thread_start_us":4281,"threads_started":1}
I20260925 05:00:47.026677 1417 tablet_service.cc:1467] Processing CreateTablet for tablet ecf1b58d911b4ef39af390985986eb9b (DEFAULT_TABLE table=test-table [id=5f3e60aa164643f5ad3777d7de35377a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:00:47.027004 1282 tablet_service.cc:1467] Processing CreateTablet for tablet ecf1b58d911b4ef39af390985986eb9b (DEFAULT_TABLE table=test-table [id=5f3e60aa164643f5ad3777d7de35377a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:00:47.027004 1137 tablet_service.cc:1467] Processing CreateTablet for tablet ecf1b58d911b4ef39af390985986eb9b (DEFAULT_TABLE table=test-table [id=5f3e60aa164643f5ad3777d7de35377a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:00:47.027953 1417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecf1b58d911b4ef39af390985986eb9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:47.028287 1137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecf1b58d911b4ef39af390985986eb9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:47.028301 1282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecf1b58d911b4ef39af390985986eb9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:47.039141 1507 tablet_bootstrap.cc:492] T ecf1b58d911b4ef39af390985986eb9b P 8463fe556af94c4ea931e4d060f9d7a5: Bootstrap starting.
I20260925 05:00:47.042302 1508 tablet_bootstrap.cc:492] T ecf1b58d911b4ef39af390985986eb9b P 105aa6bd36964ed6b923cc2e4a9e31c9: Bootstrap starting.
I20260925 05:00:47.044420 1507 tablet_bootstrap.cc:654] T ecf1b58d911b4ef39af390985986eb9b P 8463fe556af94c4ea931e4d060f9d7a5: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:47.045564 1508 tablet_bootstrap.cc:654] T ecf1b58d911b4ef39af390985986eb9b P 105aa6bd36964ed6b923cc2e4a9e31c9: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:47.045701 1507 log.cc:824] T ecf1b58d911b4ef39af390985986eb9b P 8463fe556af94c4ea931e4d060f9d7a5: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:47.046818 1508 log.cc:824] T ecf1b58d911b4ef39af390985986eb9b P 105aa6bd36964ed6b923cc2e4a9e31c9: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:47.047637 1509 tablet_bootstrap.cc:492] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206: Bootstrap starting.
I20260925 05:00:47.048882 1507 tablet_bootstrap.cc:492] T ecf1b58d911b4ef39af390985986eb9b P 8463fe556af94c4ea931e4d060f9d7a5: No bootstrap required, opened a new log
I20260925 05:00:47.049172 1507 ts_tablet_manager.cc:1397] T ecf1b58d911b4ef39af390985986eb9b P 8463fe556af94c4ea931e4d060f9d7a5: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260925 05:00:47.049345 1508 tablet_bootstrap.cc:492] T ecf1b58d911b4ef39af390985986eb9b P 105aa6bd36964ed6b923cc2e4a9e31c9: No bootstrap required, opened a new log
I20260925 05:00:47.049614 1508 ts_tablet_manager.cc:1397] T ecf1b58d911b4ef39af390985986eb9b P 105aa6bd36964ed6b923cc2e4a9e31c9: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260925 05:00:47.050967 1509 tablet_bootstrap.cc:654] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:47.052062 1509 log.cc:824] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:47.057478 1508 raft_consensus.cc:348] T ecf1b58d911b4ef39af390985986eb9b P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } }
I20260925 05:00:47.057818 1508 raft_consensus.cc:374] T ecf1b58d911b4ef39af390985986eb9b P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:47.057940 1508 raft_consensus.cc:729] T ecf1b58d911b4ef39af390985986eb9b P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 105aa6bd36964ed6b923cc2e4a9e31c9, State: Initialized, Role: FOLLOWER
I20260925 05:00:47.058154 1509 tablet_bootstrap.cc:492] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206: No bootstrap required, opened a new log
I20260925 05:00:47.058386 1509 ts_tablet_manager.cc:1397] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260925 05:00:47.058626 1508 consensus_queue.cc:260] T ecf1b58d911b4ef39af390985986eb9b P 105aa6bd36964ed6b923cc2e4a9e31c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } }
I20260925 05:00:47.061591 1347 heartbeater.cc:503] Master 127.30.247.62:35363 was elected leader, sending a full tablet report...
I20260925 05:00:47.061414 1507 raft_consensus.cc:348] T ecf1b58d911b4ef39af390985986eb9b P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } }
I20260925 05:00:47.061779 1507 raft_consensus.cc:374] T ecf1b58d911b4ef39af390985986eb9b P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:47.061865 1507 raft_consensus.cc:729] T ecf1b58d911b4ef39af390985986eb9b P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8463fe556af94c4ea931e4d060f9d7a5, State: Initialized, Role: FOLLOWER
I20260925 05:00:47.062403 1507 consensus_queue.cc:260] T ecf1b58d911b4ef39af390985986eb9b P 8463fe556af94c4ea931e4d060f9d7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } }
I20260925 05:00:47.063740 1488 heartbeater.cc:503] Master 127.30.247.62:35363 was elected leader, sending a full tablet report...
I20260925 05:00:47.064386 1507 ts_tablet_manager.cc:1428] T ecf1b58d911b4ef39af390985986eb9b P 8463fe556af94c4ea931e4d060f9d7a5: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260925 05:00:47.065379 1508 ts_tablet_manager.cc:1428] T ecf1b58d911b4ef39af390985986eb9b P 105aa6bd36964ed6b923cc2e4a9e31c9: Time spent starting tablet: real 0.016s user 0.010s sys 0.002s
I20260925 05:00:47.068262 1509 raft_consensus.cc:348] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } }
I20260925 05:00:47.068717 1509 raft_consensus.cc:374] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:47.068836 1509 raft_consensus.cc:729] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57227d8f465c4c4783262e70f0929206, State: Initialized, Role: FOLLOWER
I20260925 05:00:47.069321 1509 consensus_queue.cc:260] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } }
I20260925 05:00:47.070734 1509 ts_tablet_manager.cc:1428] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206: Time spent starting tablet: real 0.012s user 0.010s sys 0.003s
W20260925 05:00:47.164755 1209 tablet.cc:2369] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:00:47.178319 1348 tablet.cc:2369] T ecf1b58d911b4ef39af390985986eb9b P 105aa6bd36964ed6b923cc2e4a9e31c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:00:47.180116 1515 raft_consensus.cc:482] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:00:47.180439 1515 raft_consensus.cc:504] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } }
I20260925 05:00:47.182313 1515 leader_election.cc:288] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8463fe556af94c4ea931e4d060f9d7a5 (127.30.247.3:45275), 105aa6bd36964ed6b923cc2e4a9e31c9 (127.30.247.2:46397)
W20260925 05:00:47.185024 1489 tablet.cc:2369] T ecf1b58d911b4ef39af390985986eb9b P 8463fe556af94c4ea931e4d060f9d7a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:00:47.189543 1506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:47.182838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:43035 (local address 127.30.247.2:46397)
0925 05:00:47.183064 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:47.183072 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:47.183663 (+ 591us) server_negotiation.cc:407] Connection header received
0925 05:00:47.183917 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:47.183921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:47.183978 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:47.184087 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:00:47.184818 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.185593 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.187421 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.188624 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.188789 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:47.188865 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:47.189178 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 05:00:47.189301 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 05:00:47.189713 1516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:47.182577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:46397 (local address 127.30.247.1:43035)
0925 05:00:47.183458 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:47.183477 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:47.183763 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:47.184099 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:47.184104 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:47.184134 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:00:47.184655 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:47.184665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.185728 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.185735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:47.187265 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:47.187277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.188883 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.188888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:47.188995 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:47.189029 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 05:00:47.189107 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 05:00:47.189246 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":48,"spinlock_wait_cycles":7680,"thread_start_us":507,"threads_started":1}
I20260925 05:00:47.190101 1505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:47.182640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:41495 (local address 127.30.247.3:45275)
0925 05:00:47.182846 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:47.182854 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:47.183686 (+ 832us) server_negotiation.cc:407] Connection header received
0925 05:00:47.183907 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:47.183911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:47.183967 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:47.184066 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:00:47.184710 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.185950 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.187800 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.189062 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.189238 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:47.189301 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:47.189742 (+ 441us) server_negotiation.cc:299] Negotiation successful
0925 05:00:47.189879 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 05:00:47.190662 1517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:47.182627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:45275 (local address 127.30.247.1:41495)
0925 05:00:47.183569 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:47.183582 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:47.183758 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:47.184092 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:47.184096 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:47.184119 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:00:47.184504 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:47.184515 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.186099 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.186106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:47.187608 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:47.187620 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.189363 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.189372 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:47.189535 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:47.189571 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 05:00:47.190266 (+ 695us) client_negotiation.cc:240] Negotiation successful
0925 05:00:47.190405 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":23,"spinlock_wait_cycles":6912,"thread_start_us":622,"threads_started":1}
I20260925 05:00:47.190701 1302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b58d911b4ef39af390985986eb9b" candidate_uuid: "57227d8f465c4c4783262e70f0929206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" is_pre_election: true
I20260925 05:00:47.191231 1302 raft_consensus.cc:2454] T ecf1b58d911b4ef39af390985986eb9b P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57227d8f465c4c4783262e70f0929206 in term 0.
I20260925 05:00:47.191758 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b58d911b4ef39af390985986eb9b" candidate_uuid: "57227d8f465c4c4783262e70f0929206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8463fe556af94c4ea931e4d060f9d7a5" is_pre_election: true
I20260925 05:00:47.191972 1090 leader_election.cc:302] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 105aa6bd36964ed6b923cc2e4a9e31c9, 57227d8f465c4c4783262e70f0929206; no voters:
I20260925 05:00:47.192317 1442 raft_consensus.cc:2454] T ecf1b58d911b4ef39af390985986eb9b P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57227d8f465c4c4783262e70f0929206 in term 0.
I20260925 05:00:47.192374 1515 raft_consensus.cc:2781] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:00:47.192541 1515 raft_consensus.cc:482] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:00:47.192662 1515 raft_consensus.cc:3037] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:47.194885 1515 raft_consensus.cc:504] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } }
I20260925 05:00:47.195504 1515 leader_election.cc:288] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206 [CANDIDATE]: Term 1 election: Requested vote from peers 8463fe556af94c4ea931e4d060f9d7a5 (127.30.247.3:45275), 105aa6bd36964ed6b923cc2e4a9e31c9 (127.30.247.2:46397)
I20260925 05:00:47.196086 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b58d911b4ef39af390985986eb9b" candidate_uuid: "57227d8f465c4c4783262e70f0929206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8463fe556af94c4ea931e4d060f9d7a5"
I20260925 05:00:47.196071 1302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b58d911b4ef39af390985986eb9b" candidate_uuid: "57227d8f465c4c4783262e70f0929206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9"
I20260925 05:00:47.196275 1442 raft_consensus.cc:3037] T ecf1b58d911b4ef39af390985986eb9b P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:47.196276 1302 raft_consensus.cc:3037] T ecf1b58d911b4ef39af390985986eb9b P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:47.199533 1442 raft_consensus.cc:2454] T ecf1b58d911b4ef39af390985986eb9b P 8463fe556af94c4ea931e4d060f9d7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57227d8f465c4c4783262e70f0929206 in term 1.
I20260925 05:00:47.199533 1302 raft_consensus.cc:2454] T ecf1b58d911b4ef39af390985986eb9b P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57227d8f465c4c4783262e70f0929206 in term 1.
I20260925 05:00:47.200073 1090 leader_election.cc:302] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 105aa6bd36964ed6b923cc2e4a9e31c9, 57227d8f465c4c4783262e70f0929206, 8463fe556af94c4ea931e4d060f9d7a5; no voters:
I20260925 05:00:47.200388 1515 raft_consensus.cc:2781] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:00:47.200896 1515 raft_consensus.cc:686] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206 [term 1 LEADER]: Becoming Leader. State: Replica: 57227d8f465c4c4783262e70f0929206, State: Running, Role: LEADER
I20260925 05:00:47.201325 1515 consensus_queue.cc:237] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } }
I20260925 05:00:47.205932 997 catalog_manager.cc:5495] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57227d8f465c4c4783262e70f0929206 (127.30.247.1). New cstate: current_term: 1 leader_uuid: "57227d8f465c4c4783262e70f0929206" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } health_report { overall_health: UNKNOWN } } }
Sep 25 05:00:47 dist-test-slave-5n86 krb5kdc[953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312445, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:00:47.287019 1095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:47.274725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38182 (local address 127.30.247.62:35363)
0925 05:00:47.274878 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:47.274882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:47.275391 (+ 509us) server_negotiation.cc:407] Connection header received
0925 05:00:47.275577 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:47.275582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:47.275650 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:47.275748 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:47.276892 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.277598 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.278829 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.279269 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.279399 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:47.282535 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:47.282540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:47.282543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:47.282572 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:47.284820 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:47.285438 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:47.285442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:47.285444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:47.285564 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:47.285872 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:47.285875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:47.285877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:47.286114 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:47.286217 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:47.286573 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 05:00:47.286771 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":51}
I20260925 05:00:47.287022 1525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:47.274777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:35363 (local address 127.0.0.1:38182)
0925 05:00:47.275295 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:47.275311 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:47.275465 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:47.275768 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:47.275772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:47.276449 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:47.276758 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:47.276769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.277730 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.277738 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:47.278698 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:47.278708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.279436 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.279442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:47.279592 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:47.280247 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:47.280264 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:47.282377 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:47.284945 (+ 2568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:47.284954 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:47.284959 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:47.285344 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:47.285650 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:47.285653 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:47.285656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:47.285789 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:47.286225 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:47.286230 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:47.286359 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 05:00:47.286488 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 05:00:47.286663 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":280,"threads_started":1}
I20260925 05:00:47.291575 997 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:38182:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 05:00:47.292611 997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:00:47.298668 1137 tablet_service.cc:1467] Processing CreateTablet for tablet affdc6b2cd244d9eadd4aaec0bfb90e0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6559052aca964da5b807a07b1aa4085c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:00:47.298960 1282 tablet_service.cc:1467] Processing CreateTablet for tablet affdc6b2cd244d9eadd4aaec0bfb90e0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6559052aca964da5b807a07b1aa4085c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:00:47.298981 1417 tablet_service.cc:1467] Processing CreateTablet for tablet affdc6b2cd244d9eadd4aaec0bfb90e0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6559052aca964da5b807a07b1aa4085c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:00:47.299695 1137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet affdc6b2cd244d9eadd4aaec0bfb90e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:47.299757 1282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet affdc6b2cd244d9eadd4aaec0bfb90e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:47.299839 1417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet affdc6b2cd244d9eadd4aaec0bfb90e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:47.304612 1507 tablet_bootstrap.cc:492] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 8463fe556af94c4ea931e4d060f9d7a5: Bootstrap starting.
I20260925 05:00:47.306164 1507 tablet_bootstrap.cc:654] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 8463fe556af94c4ea931e4d060f9d7a5: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:47.306720 1508 tablet_bootstrap.cc:492] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 105aa6bd36964ed6b923cc2e4a9e31c9: Bootstrap starting.
I20260925 05:00:47.307286 1509 tablet_bootstrap.cc:492] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206: Bootstrap starting.
I20260925 05:00:47.308363 1507 tablet_bootstrap.cc:492] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 8463fe556af94c4ea931e4d060f9d7a5: No bootstrap required, opened a new log
I20260925 05:00:47.308449 1508 tablet_bootstrap.cc:654] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 105aa6bd36964ed6b923cc2e4a9e31c9: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:47.308512 1507 ts_tablet_manager.cc:1397] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 8463fe556af94c4ea931e4d060f9d7a5: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260925 05:00:47.308655 1509 tablet_bootstrap.cc:654] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:47.309340 1507 raft_consensus.cc:348] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:47.309599 1507 raft_consensus.cc:374] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:47.309710 1507 raft_consensus.cc:729] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8463fe556af94c4ea931e4d060f9d7a5, State: Initialized, Role: FOLLOWER
I20260925 05:00:47.309959 1507 consensus_queue.cc:260] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 8463fe556af94c4ea931e4d060f9d7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:47.310525 1507 ts_tablet_manager.cc:1428] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 8463fe556af94c4ea931e4d060f9d7a5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:00:47.310375 1514 tablet_replica.cc:439] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 8463fe556af94c4ea931e4d060f9d7a5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } } }
I20260925 05:00:47.310727 1514 tablet_replica.cc:442] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 8463fe556af94c4ea931e4d060f9d7a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:00:47.311254 1508 tablet_bootstrap.cc:492] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 105aa6bd36964ed6b923cc2e4a9e31c9: No bootstrap required, opened a new log
I20260925 05:00:47.311452 1508 ts_tablet_manager.cc:1397] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 105aa6bd36964ed6b923cc2e4a9e31c9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 05:00:47.312016 1508 raft_consensus.cc:348] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:47.312278 1508 raft_consensus.cc:374] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:47.312373 1508 raft_consensus.cc:729] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 105aa6bd36964ed6b923cc2e4a9e31c9, State: Initialized, Role: FOLLOWER
I20260925 05:00:47.312606 1508 consensus_queue.cc:260] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 105aa6bd36964ed6b923cc2e4a9e31c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:47.313206 1508 ts_tablet_manager.cc:1428] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 105aa6bd36964ed6b923cc2e4a9e31c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:00:47.313028 1513 tablet_replica.cc:439] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 105aa6bd36964ed6b923cc2e4a9e31c9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } } }
I20260925 05:00:47.313372 1513 tablet_replica.cc:442] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 105aa6bd36964ed6b923cc2e4a9e31c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:00:47.313933 1509 tablet_bootstrap.cc:492] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206: No bootstrap required, opened a new log
I20260925 05:00:47.314102 1509 ts_tablet_manager.cc:1397] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260925 05:00:47.314677 1509 raft_consensus.cc:348] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:47.314872 1509 raft_consensus.cc:374] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:47.314957 1509 raft_consensus.cc:729] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57227d8f465c4c4783262e70f0929206, State: Initialized, Role: FOLLOWER
I20260925 05:00:47.315181 1509 consensus_queue.cc:260] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:47.315682 1509 ts_tablet_manager.cc:1428] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 05:00:47.315594 1515 tablet_replica.cc:439] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } } }
I20260925 05:00:47.315891 1515 tablet_replica.cc:442] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:00:47.336463 1515 raft_consensus.cc:482] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:00:47.336716 1515 raft_consensus.cc:504] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:47.337325 1515 leader_election.cc:288] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 105aa6bd36964ed6b923cc2e4a9e31c9 (127.30.247.2:46397), 8463fe556af94c4ea931e4d060f9d7a5 (127.30.247.3:45275)
I20260925 05:00:47.337781 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "affdc6b2cd244d9eadd4aaec0bfb90e0" candidate_uuid: "57227d8f465c4c4783262e70f0929206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8463fe556af94c4ea931e4d060f9d7a5" is_pre_election: true
I20260925 05:00:47.337805 1302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "affdc6b2cd244d9eadd4aaec0bfb90e0" candidate_uuid: "57227d8f465c4c4783262e70f0929206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" is_pre_election: true
I20260925 05:00:47.338081 1442 raft_consensus.cc:2454] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57227d8f465c4c4783262e70f0929206 in term 0.
I20260925 05:00:47.338095 1302 raft_consensus.cc:2454] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57227d8f465c4c4783262e70f0929206 in term 0.
I20260925 05:00:47.338512 1088 leader_election.cc:302] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57227d8f465c4c4783262e70f0929206, 8463fe556af94c4ea931e4d060f9d7a5; no voters:
I20260925 05:00:47.338831 1515 raft_consensus.cc:2781] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:00:47.338937 1515 raft_consensus.cc:482] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:00:47.339038 1515 raft_consensus.cc:3037] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:47.341095 1515 raft_consensus.cc:504] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:47.341519 1515 leader_election.cc:288] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206 [CANDIDATE]: Term 1 election: Requested vote from peers 105aa6bd36964ed6b923cc2e4a9e31c9 (127.30.247.2:46397), 8463fe556af94c4ea931e4d060f9d7a5 (127.30.247.3:45275)
I20260925 05:00:47.341919 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "affdc6b2cd244d9eadd4aaec0bfb90e0" candidate_uuid: "57227d8f465c4c4783262e70f0929206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8463fe556af94c4ea931e4d060f9d7a5"
I20260925 05:00:47.342108 1442 raft_consensus.cc:3037] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:47.342092 1302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "affdc6b2cd244d9eadd4aaec0bfb90e0" candidate_uuid: "57227d8f465c4c4783262e70f0929206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9"
I20260925 05:00:47.342293 1302 raft_consensus.cc:3037] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:47.344007 1442 raft_consensus.cc:2454] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 8463fe556af94c4ea931e4d060f9d7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57227d8f465c4c4783262e70f0929206 in term 1.
I20260925 05:00:47.344174 1302 raft_consensus.cc:2454] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57227d8f465c4c4783262e70f0929206 in term 1.
I20260925 05:00:47.344507 1088 leader_election.cc:302] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57227d8f465c4c4783262e70f0929206, 8463fe556af94c4ea931e4d060f9d7a5; no voters:
I20260925 05:00:47.344800 1515 raft_consensus.cc:2781] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:00:47.344978 1515 raft_consensus.cc:686] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206 [term 1 LEADER]: Becoming Leader. State: Replica: 57227d8f465c4c4783262e70f0929206, State: Running, Role: LEADER
I20260925 05:00:47.345227 1515 consensus_queue.cc:237] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:47.346823 1518 tablet_replica.cc:439] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206: TxnStatusTablet state changed. Reason: New leader 57227d8f465c4c4783262e70f0929206. Latest consensus state: current_term: 1 leader_uuid: "57227d8f465c4c4783262e70f0929206" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } } }
I20260925 05:00:47.347205 1518 tablet_replica.cc:442] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:00:47.347915 1530 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 05:00:47.348184 1530 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 05:00:47.348196 997 catalog_manager.cc:5495] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57227d8f465c4c4783262e70f0929206 (127.30.247.1). New cstate: current_term: 1 leader_uuid: "57227d8f465c4c4783262e70f0929206" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } health_report { overall_health: UNKNOWN } } }
I20260925 05:00:47.370600 1503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:47.364737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57854 (local address 127.30.247.1:44329)
0925 05:00:47.364926 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:47.364933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:47.365138 (+ 205us) server_negotiation.cc:407] Connection header received
0925 05:00:47.365347 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:47.365353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:47.365464 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:47.365625 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:00:47.365925 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.367037 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.368399 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.368880 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.369028 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:47.369504 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:00:47.369983 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:00:47.370137 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:47.370229 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 05:00:47.370306 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 05:00:47.370628 1525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:47.364758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:44329 (local address 127.0.0.1:57854)
0925 05:00:47.365049 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:47.365064 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:47.365177 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:47.365635 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:47.365639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:47.365648 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:00:47.365802 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:47.365809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.367191 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.367195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:47.368249 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:47.368265 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.369057 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.369062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:47.369239 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:47.369288 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:00:47.370096 (+ 808us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:00:47.370117 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 05:00:47.370201 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 05:00:47.370348 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260925 05:00:47.378014 1302 raft_consensus.cc:1261] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 1 FOLLOWER]: Refusing update from remote peer 57227d8f465c4c4783262e70f0929206: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:00:47.378014 1442 raft_consensus.cc:1261] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 8463fe556af94c4ea931e4d060f9d7a5 [term 1 FOLLOWER]: Refusing update from remote peer 57227d8f465c4c4783262e70f0929206: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:00:47.378957 1518 consensus_queue.cc:1035] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:00:47.379299 1515 consensus_queue.cc:1035] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206 [LEADER]: Connected to new peer: Peer: permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:00:47.386463 1513 tablet_replica.cc:439] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 105aa6bd36964ed6b923cc2e4a9e31c9: TxnStatusTablet state changed. Reason: New leader 57227d8f465c4c4783262e70f0929206. Latest consensus state: current_term: 1 leader_uuid: "57227d8f465c4c4783262e70f0929206" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } } }
I20260925 05:00:47.386739 1513 tablet_replica.cc:442] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 105aa6bd36964ed6b923cc2e4a9e31c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:00:47.389370 1533 mvcc.cc:205] Tried to move back new op lower bound from 7333119784450752512 to 7333119784327864320. Current Snapshot: MvccSnapshot[applied={T|T < 7333119784450752512}]
I20260925 05:00:47.390568 1534 mvcc.cc:205] Tried to move back new op lower bound from 7333119784450752512 to 7333119784327864320. Current Snapshot: MvccSnapshot[applied={T|T < 7333119784450752512}]
I20260925 05:00:47.393889 1518 tablet_replica.cc:439] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57227d8f465c4c4783262e70f0929206" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } } }
I20260925 05:00:47.394132 1518 tablet_replica.cc:442] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:00:47.398231 1513 tablet_replica.cc:439] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 105aa6bd36964ed6b923cc2e4a9e31c9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57227d8f465c4c4783262e70f0929206" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } } }
I20260925 05:00:47.398451 1513 tablet_replica.cc:442] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 105aa6bd36964ed6b923cc2e4a9e31c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:00:47.399247 1514 tablet_replica.cc:439] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 8463fe556af94c4ea931e4d060f9d7a5: TxnStatusTablet state changed. Reason: New leader 57227d8f465c4c4783262e70f0929206. Latest consensus state: current_term: 1 leader_uuid: "57227d8f465c4c4783262e70f0929206" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } } }
I20260925 05:00:47.399482 1514 tablet_replica.cc:442] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 8463fe556af94c4ea931e4d060f9d7a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:00:47.402086 1515 tablet_replica.cc:439] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57227d8f465c4c4783262e70f0929206" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } } }
I20260925 05:00:47.402410 1515 tablet_replica.cc:442] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 57227d8f465c4c4783262e70f0929206: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:00:47.421187 1514 tablet_replica.cc:439] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 8463fe556af94c4ea931e4d060f9d7a5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57227d8f465c4c4783262e70f0929206" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } } }
I20260925 05:00:47.421412 1514 tablet_replica.cc:442] T affdc6b2cd244d9eadd4aaec0bfb90e0 P 8463fe556af94c4ea931e4d060f9d7a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:00:47.424396 1503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:47.416644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57868 (local address 127.30.247.1:44329)
0925 05:00:47.416803 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:47.416810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:47.416829 (+ 19us) server_negotiation.cc:407] Connection header received
0925 05:00:47.417022 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:47.417030 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:47.417137 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:47.417433 (+ 296us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:00:47.418383 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.419967 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.422251 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.422861 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.423006 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:47.423527 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:00:47.423664 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:00:47.423755 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:47.424005 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 05:00:47.424090 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 05:00:47.449908 1503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:47.442575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:35023 (local address 127.30.247.1:44329)
0925 05:00:47.443038 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:47.443044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:47.443497 (+ 453us) server_negotiation.cc:407] Connection header received
0925 05:00:47.443657 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:47.443662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:47.443744 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:47.443833 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:00:47.444559 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.445652 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.447750 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.448448 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.448602 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:47.449148 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:00:47.449284 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:00:47.449372 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:47.449501 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 05:00:47.449579 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329}
I20260925 05:00:47.450179 1545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:47.442657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:44329 (local address 127.30.247.1:35023)
0925 05:00:47.443384 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:47.443407 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:47.443541 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:47.443986 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:47.443992 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:47.444008 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:00:47.444380 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:47.444393 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.445783 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.445788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:47.447604 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:47.447619 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.448683 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.448690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:47.448868 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:47.448933 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:00:47.449375 (+ 442us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:00:47.449386 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 05:00:47.449467 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 05:00:47.449631 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":403,"threads_started":1}
I20260925 05:00:47.459013 1442 raft_consensus.cc:1261] T ecf1b58d911b4ef39af390985986eb9b P 8463fe556af94c4ea931e4d060f9d7a5 [term 1 FOLLOWER]: Refusing update from remote peer 57227d8f465c4c4783262e70f0929206: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:00:47.459053 1302 raft_consensus.cc:1261] T ecf1b58d911b4ef39af390985986eb9b P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 1 FOLLOWER]: Refusing update from remote peer 57227d8f465c4c4783262e70f0929206: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:00:47.459528 1540 consensus_queue.cc:1035] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:00:47.459901 1518 consensus_queue.cc:1035] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206 [LEADER]: Connected to new peer: Peer: permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:00:47.511739 997 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:00:47.512019 997 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:00:47.515545 997 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e75816c69a345df9807f8da66ce85dd: Sending DeleteTablet for 3 replicas of tablet ecf1b58d911b4ef39af390985986eb9b
I20260925 05:00:47.516669 1417 tablet_service.cc:1514] Processing DeleteTablet for tablet ecf1b58d911b4ef39af390985986eb9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:00:47 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:38176
I20260925 05:00:47.516738 1282 tablet_service.cc:1514] Processing DeleteTablet for tablet ecf1b58d911b4ef39af390985986eb9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:00:47 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:48826
I20260925 05:00:47.517805 1136 tablet_service.cc:1514] Processing DeleteTablet for tablet ecf1b58d911b4ef39af390985986eb9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:00:47 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:57844
I20260925 05:00:47.518405 1549 tablet_replica.cc:330] T ecf1b58d911b4ef39af390985986eb9b P 8463fe556af94c4ea931e4d060f9d7a5: stopping tablet replica
I20260925 05:00:47.518304 998 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:00:47.518926 998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:00:47.518921 1549 raft_consensus.cc:2229] T ecf1b58d911b4ef39af390985986eb9b P 8463fe556af94c4ea931e4d060f9d7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:00:47.519285 1549 raft_consensus.cc:2258] T ecf1b58d911b4ef39af390985986eb9b P 8463fe556af94c4ea931e4d060f9d7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:00:47.527886 1551 tablet_replica.cc:330] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206: stopping tablet replica
I20260925 05:00:47.528220 1549 ts_tablet_manager.cc:1905] T ecf1b58d911b4ef39af390985986eb9b P 8463fe556af94c4ea931e4d060f9d7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:00:47.528595 1277 tablet_service.cc:1467] Processing CreateTablet for tablet b2d34f012ce04b359a7a5a235c3c0d0d (DEFAULT_TABLE table=test-table [id=90b3f8ca1e734743b976eabf06722a97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:00:47.529194 1277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2d34f012ce04b359a7a5a235c3c0d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:47.529593 1551 raft_consensus.cc:2229] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:00:47.530501 1551 raft_consensus.cc:2258] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:00:47.531518 1550 tablet_replica.cc:330] T ecf1b58d911b4ef39af390985986eb9b P 105aa6bd36964ed6b923cc2e4a9e31c9: stopping tablet replica
I20260925 05:00:47.532213 1550 raft_consensus.cc:2229] T ecf1b58d911b4ef39af390985986eb9b P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:00:47.532917 1417 tablet_service.cc:1467] Processing CreateTablet for tablet b2d34f012ce04b359a7a5a235c3c0d0d (DEFAULT_TABLE table=test-table [id=90b3f8ca1e734743b976eabf06722a97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:00:47.533373 1417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2d34f012ce04b359a7a5a235c3c0d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:47.533744 1550 raft_consensus.cc:2258] T ecf1b58d911b4ef39af390985986eb9b P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:00:47.534564 1551 ts_tablet_manager.cc:1905] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:00:47.536258 1549 ts_tablet_manager.cc:1918] T ecf1b58d911b4ef39af390985986eb9b P 8463fe556af94c4ea931e4d060f9d7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:00:47.536404 1549 log.cc:1196] T ecf1b58d911b4ef39af390985986eb9b P 8463fe556af94c4ea931e4d060f9d7a5: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-2/wal/wals/ecf1b58d911b4ef39af390985986eb9b
I20260925 05:00:47.537539 1136 tablet_service.cc:1467] Processing CreateTablet for tablet b2d34f012ce04b359a7a5a235c3c0d0d (DEFAULT_TABLE table=test-table [id=90b3f8ca1e734743b976eabf06722a97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:00:47.538010 1136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2d34f012ce04b359a7a5a235c3c0d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:47.539217 1550 ts_tablet_manager.cc:1905] T ecf1b58d911b4ef39af390985986eb9b P 105aa6bd36964ed6b923cc2e4a9e31c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:00:47.544871 1509 tablet_bootstrap.cc:492] T b2d34f012ce04b359a7a5a235c3c0d0d P 57227d8f465c4c4783262e70f0929206: Bootstrap starting.
I20260925 05:00:47.546685 1509 tablet_bootstrap.cc:654] T b2d34f012ce04b359a7a5a235c3c0d0d P 57227d8f465c4c4783262e70f0929206: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:47.549397 1551 ts_tablet_manager.cc:1918] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:00:47.549643 1507 tablet_bootstrap.cc:492] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5: Bootstrap starting.
I20260925 05:00:47.551219 1507 tablet_bootstrap.cc:654] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:47.552209 1550 ts_tablet_manager.cc:1918] T ecf1b58d911b4ef39af390985986eb9b P 105aa6bd36964ed6b923cc2e4a9e31c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:00:47.552459 1551 log.cc:1196] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-0/wal/wals/ecf1b58d911b4ef39af390985986eb9b
I20260925 05:00:47.552788 1549 ts_tablet_manager.cc:1939] T ecf1b58d911b4ef39af390985986eb9b P 8463fe556af94c4ea931e4d060f9d7a5: Deleting consensus metadata
I20260925 05:00:47.553213 1551 ts_tablet_manager.cc:1939] T ecf1b58d911b4ef39af390985986eb9b P 57227d8f465c4c4783262e70f0929206: Deleting consensus metadata
I20260925 05:00:47.553478 1550 log.cc:1196] T ecf1b58d911b4ef39af390985986eb9b P 105aa6bd36964ed6b923cc2e4a9e31c9: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-1/wal/wals/ecf1b58d911b4ef39af390985986eb9b
I20260925 05:00:47.554615 983 catalog_manager.cc:4841] TS 8463fe556af94c4ea931e4d060f9d7a5 (127.30.247.3:45275): tablet ecf1b58d911b4ef39af390985986eb9b (table test-table [id=5f3e60aa164643f5ad3777d7de35377a]) successfully deleted
I20260925 05:00:47.555250 1550 ts_tablet_manager.cc:1939] T ecf1b58d911b4ef39af390985986eb9b P 105aa6bd36964ed6b923cc2e4a9e31c9: Deleting consensus metadata
I20260925 05:00:47.555986 1508 tablet_bootstrap.cc:492] T b2d34f012ce04b359a7a5a235c3c0d0d P 105aa6bd36964ed6b923cc2e4a9e31c9: Bootstrap starting.
I20260925 05:00:47.556896 982 catalog_manager.cc:4841] TS 57227d8f465c4c4783262e70f0929206 (127.30.247.1:44329): tablet ecf1b58d911b4ef39af390985986eb9b (table test-table [id=5f3e60aa164643f5ad3777d7de35377a]) successfully deleted
I20260925 05:00:47.557583 1508 tablet_bootstrap.cc:654] T b2d34f012ce04b359a7a5a235c3c0d0d P 105aa6bd36964ed6b923cc2e4a9e31c9: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:47.558758 1509 tablet_bootstrap.cc:492] T b2d34f012ce04b359a7a5a235c3c0d0d P 57227d8f465c4c4783262e70f0929206: No bootstrap required, opened a new log
I20260925 05:00:47.558786 1507 tablet_bootstrap.cc:492] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5: No bootstrap required, opened a new log
I20260925 05:00:47.558928 1507 ts_tablet_manager.cc:1397] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260925 05:00:47.558930 985 catalog_manager.cc:4841] TS 105aa6bd36964ed6b923cc2e4a9e31c9 (127.30.247.2:46397): tablet ecf1b58d911b4ef39af390985986eb9b (table test-table [id=5f3e60aa164643f5ad3777d7de35377a]) successfully deleted
I20260925 05:00:47.559090 1509 ts_tablet_manager.cc:1397] T b2d34f012ce04b359a7a5a235c3c0d0d P 57227d8f465c4c4783262e70f0929206: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.001s
I20260925 05:00:47.559659 1507 raft_consensus.cc:348] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:47.559916 1507 raft_consensus.cc:374] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:47.559999 1507 raft_consensus.cc:729] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8463fe556af94c4ea931e4d060f9d7a5, State: Initialized, Role: FOLLOWER
I20260925 05:00:47.560282 1507 consensus_queue.cc:260] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:47.560890 1507 ts_tablet_manager.cc:1428] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:00:47.560422 1509 raft_consensus.cc:348] T b2d34f012ce04b359a7a5a235c3c0d0d P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:47.561390 1509 raft_consensus.cc:374] T b2d34f012ce04b359a7a5a235c3c0d0d P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:47.561473 1509 raft_consensus.cc:729] T b2d34f012ce04b359a7a5a235c3c0d0d P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57227d8f465c4c4783262e70f0929206, State: Initialized, Role: FOLLOWER
I20260925 05:00:47.561697 1509 consensus_queue.cc:260] T b2d34f012ce04b359a7a5a235c3c0d0d P 57227d8f465c4c4783262e70f0929206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:47.562302 1509 ts_tablet_manager.cc:1428] T b2d34f012ce04b359a7a5a235c3c0d0d P 57227d8f465c4c4783262e70f0929206: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260925 05:00:47.569420 1508 tablet_bootstrap.cc:492] T b2d34f012ce04b359a7a5a235c3c0d0d P 105aa6bd36964ed6b923cc2e4a9e31c9: No bootstrap required, opened a new log
I20260925 05:00:47.569584 1508 ts_tablet_manager.cc:1397] T b2d34f012ce04b359a7a5a235c3c0d0d P 105aa6bd36964ed6b923cc2e4a9e31c9: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.006s
I20260925 05:00:47.570303 1508 raft_consensus.cc:348] T b2d34f012ce04b359a7a5a235c3c0d0d P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:47.570537 1508 raft_consensus.cc:374] T b2d34f012ce04b359a7a5a235c3c0d0d P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:47.570614 1508 raft_consensus.cc:729] T b2d34f012ce04b359a7a5a235c3c0d0d P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 105aa6bd36964ed6b923cc2e4a9e31c9, State: Initialized, Role: FOLLOWER
I20260925 05:00:47.570891 1508 consensus_queue.cc:260] T b2d34f012ce04b359a7a5a235c3c0d0d P 105aa6bd36964ed6b923cc2e4a9e31c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:47.571461 1508 ts_tablet_manager.cc:1428] T b2d34f012ce04b359a7a5a235c3c0d0d P 105aa6bd36964ed6b923cc2e4a9e31c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 05:00:47.585619 1514 raft_consensus.cc:482] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:00:47.585839 1514 raft_consensus.cc:504] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:47.587345 1514 leader_election.cc:288] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57227d8f465c4c4783262e70f0929206 (127.30.247.1:44329), 105aa6bd36964ed6b923cc2e4a9e31c9 (127.30.247.2:46397)
I20260925 05:00:47.595191 1503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:47.588243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:46043 (local address 127.30.247.1:44329)
0925 05:00:47.588406 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:47.588414 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:47.588448 (+ 34us) server_negotiation.cc:407] Connection header received
0925 05:00:47.588815 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:47.588826 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:47.588913 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:47.588983 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:00:47.589674 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.590586 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.592865 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.594199 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.594355 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:47.594435 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:47.594766 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 05:00:47.594893 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 05:00:47.595328 1506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:47.587772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:43355 (local address 127.30.247.2:46397)
0925 05:00:47.587957 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:47.587965 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:47.589249 (+ 1284us) server_negotiation.cc:407] Connection header received
0925 05:00:47.589398 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:47.589404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:47.589475 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:47.589570 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:00:47.590139 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.591228 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.593400 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.594215 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.594383 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:47.594451 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:47.594929 (+ 478us) server_negotiation.cc:299] Negotiation successful
0925 05:00:47.595054 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 05:00:47.595239 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:47.587474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:44329 (local address 127.30.247.3:46043)
0925 05:00:47.588363 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:47.588386 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:47.588507 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:47.589187 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:47.589194 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:47.589224 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:00:47.589493 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:47.589501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.590733 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.590736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:47.592668 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:47.592681 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.594390 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.594396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:47.594593 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:47.594631 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 05:00:47.594725 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 05:00:47.594853 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"spinlock_wait_cycles":3968,"thread_start_us":1364,"threads_started":1}
I20260925 05:00:47.596011 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:47.587700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:46397 (local address 127.30.247.3:43355)
0925 05:00:47.589154 (+ 1454us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:47.589171 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:47.589269 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:47.589597 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:47.589601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:47.589629 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:00:47.589963 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:47.589976 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.591370 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.591375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:47.593266 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:47.593277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.594600 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.594610 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:47.594769 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:47.594796 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 05:00:47.595534 (+ 738us) client_negotiation.cc:240] Negotiation successful
0925 05:00:47.595677 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1315,"mutex_wait_us":4,"thread_start_us":844,"threads_started":1}
I20260925 05:00:47.596194 1162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d34f012ce04b359a7a5a235c3c0d0d" candidate_uuid: "8463fe556af94c4ea931e4d060f9d7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57227d8f465c4c4783262e70f0929206" is_pre_election: true
I20260925 05:00:47.596630 1301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d34f012ce04b359a7a5a235c3c0d0d" candidate_uuid: "8463fe556af94c4ea931e4d060f9d7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" is_pre_election: true
I20260925 05:00:47.596657 1162 raft_consensus.cc:2454] T b2d34f012ce04b359a7a5a235c3c0d0d P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8463fe556af94c4ea931e4d060f9d7a5 in term 0.
I20260925 05:00:47.596812 1301 raft_consensus.cc:2454] T b2d34f012ce04b359a7a5a235c3c0d0d P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8463fe556af94c4ea931e4d060f9d7a5 in term 0.
I20260925 05:00:47.597254 1371 leader_election.cc:302] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 105aa6bd36964ed6b923cc2e4a9e31c9, 57227d8f465c4c4783262e70f0929206, 8463fe556af94c4ea931e4d060f9d7a5; no voters:
I20260925 05:00:47.597664 1514 raft_consensus.cc:2781] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:00:47.597795 1514 raft_consensus.cc:482] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:00:47.597916 1514 raft_consensus.cc:3037] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:47.600234 1514 raft_consensus.cc:504] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:47.600847 1514 leader_election.cc:288] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5 [CANDIDATE]: Term 1 election: Requested vote from peers 57227d8f465c4c4783262e70f0929206 (127.30.247.1:44329), 105aa6bd36964ed6b923cc2e4a9e31c9 (127.30.247.2:46397)
I20260925 05:00:47.601244 1162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d34f012ce04b359a7a5a235c3c0d0d" candidate_uuid: "8463fe556af94c4ea931e4d060f9d7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57227d8f465c4c4783262e70f0929206"
I20260925 05:00:47.601430 1162 raft_consensus.cc:3037] T b2d34f012ce04b359a7a5a235c3c0d0d P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:47.603565 1162 raft_consensus.cc:2454] T b2d34f012ce04b359a7a5a235c3c0d0d P 57227d8f465c4c4783262e70f0929206 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8463fe556af94c4ea931e4d060f9d7a5 in term 1.
I20260925 05:00:47.603878 1301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d34f012ce04b359a7a5a235c3c0d0d" candidate_uuid: "8463fe556af94c4ea931e4d060f9d7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9"
I20260925 05:00:47.603945 1368 leader_election.cc:302] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57227d8f465c4c4783262e70f0929206, 8463fe556af94c4ea931e4d060f9d7a5; no voters:
I20260925 05:00:47.604087 1301 raft_consensus.cc:3037] T b2d34f012ce04b359a7a5a235c3c0d0d P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:47.604316 1514 raft_consensus.cc:2781] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:00:47.607892 1301 raft_consensus.cc:2454] T b2d34f012ce04b359a7a5a235c3c0d0d P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8463fe556af94c4ea931e4d060f9d7a5 in term 1.
I20260925 05:00:47.608414 1514 raft_consensus.cc:686] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5 [term 1 LEADER]: Becoming Leader. State: Replica: 8463fe556af94c4ea931e4d060f9d7a5, State: Running, Role: LEADER
I20260925 05:00:47.608839 1514 consensus_queue.cc:237] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:47.612046 998 catalog_manager.cc:5495] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8463fe556af94c4ea931e4d060f9d7a5 (127.30.247.3). New cstate: current_term: 1 leader_uuid: "8463fe556af94c4ea931e4d060f9d7a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } health_report { overall_health: HEALTHY } } }
I20260925 05:00:47.638603 1505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:47.632702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38180 (local address 127.30.247.3:45275)
0925 05:00:47.632878 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:47.632885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:47.633049 (+ 164us) server_negotiation.cc:407] Connection header received
0925 05:00:47.633273 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:47.633276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:47.633353 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:47.633473 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:00:47.633798 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.634521 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.636336 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.636767 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.636864 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:47.637392 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:00:47.637897 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:00:47.638104 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:47.638198 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 05:00:47.638260 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 05:00:47.653582 1503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:47.648089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:38075 (local address 127.30.247.1:44329)
0925 05:00:47.648250 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:47.648254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:47.648850 (+ 596us) server_negotiation.cc:407] Connection header received
0925 05:00:47.649041 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:47.649046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:47.649123 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:47.649217 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:00:47.649680 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.650452 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.651783 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.652279 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.652422 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:47.652864 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:00:47.652968 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:00:47.653059 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:47.653211 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 05:00:47.653284 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 05:00:47.653651 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:47.648129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:44329 (local address 127.30.247.3:38075)
0925 05:00:47.648744 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:47.648759 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:47.648915 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:47.649231 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:47.649236 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:47.649247 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:00:47.649543 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:47.649551 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.650577 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.650586 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:47.651641 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:47.651651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.652450 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.652457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:47.652635 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:47.652686 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:00:47.653061 (+ 375us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:00:47.653074 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 05:00:47.653185 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 05:00:47.653305 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":314,"threads_started":1}
I20260925 05:00:47.662976 1301 raft_consensus.cc:1261] T b2d34f012ce04b359a7a5a235c3c0d0d P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 1 FOLLOWER]: Refusing update from remote peer 8463fe556af94c4ea931e4d060f9d7a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:00:47.663612 1162 raft_consensus.cc:1261] T b2d34f012ce04b359a7a5a235c3c0d0d P 57227d8f465c4c4783262e70f0929206 [term 1 FOLLOWER]: Refusing update from remote peer 8463fe556af94c4ea931e4d060f9d7a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:00:47.664340 1557 consensus_queue.cc:1035] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:00:47.664742 1514 consensus_queue.cc:1035] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:00:47.673898 1535 mvcc.cc:205] Tried to move back new op lower bound from 7333119785621983232 to 7333119785409241088. Current Snapshot: MvccSnapshot[applied={T|T < 7333119785621983232}]
I20260925 05:00:47.696827 1545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:47.691417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:45275 (local address 127.30.247.1:57715)
0925 05:00:47.691629 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:47.691651 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:47.691761 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:47.692115 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:47.692119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:47.692130 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:00:47.692318 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:47.692327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.693405 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.693411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:47.694899 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:47.694911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.695615 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.695621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:47.695812 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:47.695871 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:00:47.696314 (+ 443us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:00:47.696326 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 05:00:47.696424 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 05:00:47.696599 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 05:00:47.696815 1505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:47.691703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:57715 (local address 127.30.247.3:45275)
0925 05:00:47.691861 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:47.691867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:47.691883 (+ 16us) server_negotiation.cc:407] Connection header received
0925 05:00:47.691952 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:47.691956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:47.692008 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:47.692100 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:00:47.692442 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.693272 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.695048 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:47.695460 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:47.695605 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:47.696050 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:00:47.696187 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:00:47.696289 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:47.696458 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 05:00:47.696557 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 05:00:47.698313 1535 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 05:00:49.286728 997 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:00:49.286978 997 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:00:49.290879 997 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e75816c69a345df9807f8da66ce85dd: Sending DeleteTablet for 3 replicas of tablet b2d34f012ce04b359a7a5a235c3c0d0d
I20260925 05:00:49.291544 1136 tablet_service.cc:1514] Processing DeleteTablet for tablet b2d34f012ce04b359a7a5a235c3c0d0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:00:49 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:57844
I20260925 05:00:49.291730 1417 tablet_service.cc:1514] Processing DeleteTablet for tablet b2d34f012ce04b359a7a5a235c3c0d0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:00:49 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:38176
I20260925 05:00:49.291746 1277 tablet_service.cc:1514] Processing DeleteTablet for tablet b2d34f012ce04b359a7a5a235c3c0d0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:00:49 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:48826
I20260925 05:00:49.303892 1589 tablet_replica.cc:330] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5: stopping tablet replica
I20260925 05:00:49.304093 1589 raft_consensus.cc:2229] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:00:49.304653 1589 raft_consensus.cc:2258] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:00:49.305048 1590 tablet_replica.cc:330] T b2d34f012ce04b359a7a5a235c3c0d0d P 105aa6bd36964ed6b923cc2e4a9e31c9: stopping tablet replica
I20260925 05:00:49.305150 1588 tablet_replica.cc:330] T b2d34f012ce04b359a7a5a235c3c0d0d P 57227d8f465c4c4783262e70f0929206: stopping tablet replica
I20260925 05:00:49.305369 1590 raft_consensus.cc:2229] T b2d34f012ce04b359a7a5a235c3c0d0d P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:00:49.305382 1588 raft_consensus.cc:2229] T b2d34f012ce04b359a7a5a235c3c0d0d P 57227d8f465c4c4783262e70f0929206 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:00:49.305586 1588 raft_consensus.cc:2258] T b2d34f012ce04b359a7a5a235c3c0d0d P 57227d8f465c4c4783262e70f0929206 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:00:49.305707 1590 raft_consensus.cc:2258] T b2d34f012ce04b359a7a5a235c3c0d0d P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:00:49.305841 1589 ts_tablet_manager.cc:1905] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:00:49.306583 1588 ts_tablet_manager.cc:1905] T b2d34f012ce04b359a7a5a235c3c0d0d P 57227d8f465c4c4783262e70f0929206: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:00:49.306653 1590 ts_tablet_manager.cc:1905] T b2d34f012ce04b359a7a5a235c3c0d0d P 105aa6bd36964ed6b923cc2e4a9e31c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:00:49.312619 1589 ts_tablet_manager.cc:1918] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 05:00:49.312763 1589 log.cc:1196] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-2/wal/wals/b2d34f012ce04b359a7a5a235c3c0d0d
I20260925 05:00:49.313201 1590 ts_tablet_manager.cc:1918] T b2d34f012ce04b359a7a5a235c3c0d0d P 105aa6bd36964ed6b923cc2e4a9e31c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 05:00:49.313225 1589 ts_tablet_manager.cc:1939] T b2d34f012ce04b359a7a5a235c3c0d0d P 8463fe556af94c4ea931e4d060f9d7a5: Deleting consensus metadata
I20260925 05:00:49.313441 1590 log.cc:1196] T b2d34f012ce04b359a7a5a235c3c0d0d P 105aa6bd36964ed6b923cc2e4a9e31c9: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-1/wal/wals/b2d34f012ce04b359a7a5a235c3c0d0d
I20260925 05:00:49.313831 1588 ts_tablet_manager.cc:1918] T b2d34f012ce04b359a7a5a235c3c0d0d P 57227d8f465c4c4783262e70f0929206: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 05:00:49.313987 1588 log.cc:1196] T b2d34f012ce04b359a7a5a235c3c0d0d P 57227d8f465c4c4783262e70f0929206: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-0/wal/wals/b2d34f012ce04b359a7a5a235c3c0d0d
I20260925 05:00:49.313982 1590 ts_tablet_manager.cc:1939] T b2d34f012ce04b359a7a5a235c3c0d0d P 105aa6bd36964ed6b923cc2e4a9e31c9: Deleting consensus metadata
I20260925 05:00:49.314443 1588 ts_tablet_manager.cc:1939] T b2d34f012ce04b359a7a5a235c3c0d0d P 57227d8f465c4c4783262e70f0929206: Deleting consensus metadata
I20260925 05:00:49.314415 983 catalog_manager.cc:4841] TS 8463fe556af94c4ea931e4d060f9d7a5 (127.30.247.3:45275): tablet b2d34f012ce04b359a7a5a235c3c0d0d (table test-table [id=90b3f8ca1e734743b976eabf06722a97]) successfully deleted
I20260925 05:00:49.315197 985 catalog_manager.cc:4841] TS 105aa6bd36964ed6b923cc2e4a9e31c9 (127.30.247.2:46397): tablet b2d34f012ce04b359a7a5a235c3c0d0d (table test-table [id=90b3f8ca1e734743b976eabf06722a97]) successfully deleted
I20260925 05:00:49.315445 982 catalog_manager.cc:4841] TS 57227d8f465c4c4783262e70f0929206 (127.30.247.1:44329): tablet b2d34f012ce04b359a7a5a235c3c0d0d (table test-table [id=90b3f8ca1e734743b976eabf06722a97]) successfully deleted
Sep 25 05:00:49 dist-test-slave-5n86 krb5kdc[953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312449, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 05:00:49 dist-test-slave-5n86 krb5kdc[953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312449, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:00:49.352146 1598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:49.338437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40776 (local address 127.30.247.62:35363)
0925 05:00:49.339026 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:49.339034 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:49.339078 (+ 44us) server_negotiation.cc:407] Connection header received
0925 05:00:49.339306 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:49.339312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:49.339416 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:49.339552 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:49.340963 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:49.341733 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:49.342925 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:49.343353 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:49.343469 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:49.347087 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:49.347091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:49.347095 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:49.347125 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:49.349675 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:49.350288 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:49.350294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:49.350302 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:49.350440 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:49.350782 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:49.350798 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:49.350800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:49.351061 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:49.351297 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:49.351586 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 05:00:49.351806 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":284,"threads_started":1}
I20260925 05:00:49.355417 997 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:00:49.355968 997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:00:49.362061 1136 tablet_service.cc:1467] Processing CreateTablet for tablet 5dd3fbc84a7044e4ab4afe2094081816 (DEFAULT_TABLE table=test-table [id=57937c5059d7421da8a9c0dbc3ffbecc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:00:49.362210 1277 tablet_service.cc:1467] Processing CreateTablet for tablet 5dd3fbc84a7044e4ab4afe2094081816 (DEFAULT_TABLE table=test-table [id=57937c5059d7421da8a9c0dbc3ffbecc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:00:49.362504 1136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dd3fbc84a7044e4ab4afe2094081816. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:49.362429 1417 tablet_service.cc:1467] Processing CreateTablet for tablet 5dd3fbc84a7044e4ab4afe2094081816 (DEFAULT_TABLE table=test-table [id=57937c5059d7421da8a9c0dbc3ffbecc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:00:49.362633 1277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dd3fbc84a7044e4ab4afe2094081816. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:49.362818 1417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dd3fbc84a7044e4ab4afe2094081816. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:49.368839 1599 tablet_bootstrap.cc:492] T 5dd3fbc84a7044e4ab4afe2094081816 P 105aa6bd36964ed6b923cc2e4a9e31c9: Bootstrap starting.
I20260925 05:00:49.369650 1601 tablet_bootstrap.cc:492] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206: Bootstrap starting.
I20260925 05:00:49.370502 1599 tablet_bootstrap.cc:654] T 5dd3fbc84a7044e4ab4afe2094081816 P 105aa6bd36964ed6b923cc2e4a9e31c9: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:49.371559 1600 tablet_bootstrap.cc:492] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5: Bootstrap starting.
I20260925 05:00:49.372794 1601 tablet_bootstrap.cc:654] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:49.373672 1600 tablet_bootstrap.cc:654] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:49.374841 1601 tablet_bootstrap.cc:492] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206: No bootstrap required, opened a new log
I20260925 05:00:49.374889 1599 tablet_bootstrap.cc:492] T 5dd3fbc84a7044e4ab4afe2094081816 P 105aa6bd36964ed6b923cc2e4a9e31c9: No bootstrap required, opened a new log
I20260925 05:00:49.375000 1601 ts_tablet_manager.cc:1397] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 05:00:49.375046 1599 ts_tablet_manager.cc:1397] T 5dd3fbc84a7044e4ab4afe2094081816 P 105aa6bd36964ed6b923cc2e4a9e31c9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260925 05:00:49.375769 1600 tablet_bootstrap.cc:492] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5: No bootstrap required, opened a new log
I20260925 05:00:49.375900 1600 ts_tablet_manager.cc:1397] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260925 05:00:49.375797 1601 raft_consensus.cc:348] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:49.375820 1599 raft_consensus.cc:348] T 5dd3fbc84a7044e4ab4afe2094081816 P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:49.376019 1601 raft_consensus.cc:374] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:49.376060 1599 raft_consensus.cc:374] T 5dd3fbc84a7044e4ab4afe2094081816 P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:49.376112 1601 raft_consensus.cc:729] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57227d8f465c4c4783262e70f0929206, State: Initialized, Role: FOLLOWER
I20260925 05:00:49.376157 1599 raft_consensus.cc:729] T 5dd3fbc84a7044e4ab4afe2094081816 P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 105aa6bd36964ed6b923cc2e4a9e31c9, State: Initialized, Role: FOLLOWER
I20260925 05:00:49.376399 1599 consensus_queue.cc:260] T 5dd3fbc84a7044e4ab4afe2094081816 P 105aa6bd36964ed6b923cc2e4a9e31c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:49.376364 1601 consensus_queue.cc:260] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:49.376680 1600 raft_consensus.cc:348] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:49.376902 1600 raft_consensus.cc:374] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:49.376993 1600 raft_consensus.cc:729] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8463fe556af94c4ea931e4d060f9d7a5, State: Initialized, Role: FOLLOWER
I20260925 05:00:49.377235 1600 consensus_queue.cc:260] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:49.377887 1600 ts_tablet_manager.cc:1428] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:00:49.379222 1601 ts_tablet_manager.cc:1428] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260925 05:00:49.380239 1599 ts_tablet_manager.cc:1428] T 5dd3fbc84a7044e4ab4afe2094081816 P 105aa6bd36964ed6b923cc2e4a9e31c9: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20260925 05:00:49.559175 1580 raft_consensus.cc:482] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:00:49.559031 1567 raft_consensus.cc:482] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:00:49.559358 1580 raft_consensus.cc:504] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:49.559358 1567 raft_consensus.cc:504] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:49.560127 1567 leader_election.cc:288] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 105aa6bd36964ed6b923cc2e4a9e31c9 (127.30.247.2:46397), 8463fe556af94c4ea931e4d060f9d7a5 (127.30.247.3:45275)
I20260925 05:00:49.560148 1580 leader_election.cc:288] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57227d8f465c4c4783262e70f0929206 (127.30.247.1:44329), 105aa6bd36964ed6b923cc2e4a9e31c9 (127.30.247.2:46397)
I20260925 05:00:49.560586 1162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd3fbc84a7044e4ab4afe2094081816" candidate_uuid: "8463fe556af94c4ea931e4d060f9d7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57227d8f465c4c4783262e70f0929206" is_pre_election: true
I20260925 05:00:49.560642 1301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd3fbc84a7044e4ab4afe2094081816" candidate_uuid: "57227d8f465c4c4783262e70f0929206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" is_pre_election: true
I20260925 05:00:49.560842 1162 raft_consensus.cc:2454] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8463fe556af94c4ea931e4d060f9d7a5 in term 0.
I20260925 05:00:49.560873 1301 raft_consensus.cc:2454] T 5dd3fbc84a7044e4ab4afe2094081816 P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57227d8f465c4c4783262e70f0929206 in term 0.
I20260925 05:00:49.560858 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd3fbc84a7044e4ab4afe2094081816" candidate_uuid: "57227d8f465c4c4783262e70f0929206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8463fe556af94c4ea931e4d060f9d7a5" is_pre_election: true
I20260925 05:00:49.560981 1302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd3fbc84a7044e4ab4afe2094081816" candidate_uuid: "8463fe556af94c4ea931e4d060f9d7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" is_pre_election: true
I20260925 05:00:49.561072 1442 raft_consensus.cc:2454] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57227d8f465c4c4783262e70f0929206 in term 0.
I20260925 05:00:49.561173 1302 raft_consensus.cc:2454] T 5dd3fbc84a7044e4ab4afe2094081816 P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8463fe556af94c4ea931e4d060f9d7a5 in term 0.
I20260925 05:00:49.561305 1090 leader_election.cc:302] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 105aa6bd36964ed6b923cc2e4a9e31c9, 57227d8f465c4c4783262e70f0929206; no voters:
I20260925 05:00:49.561300 1368 leader_election.cc:302] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57227d8f465c4c4783262e70f0929206, 8463fe556af94c4ea931e4d060f9d7a5; no voters:
I20260925 05:00:49.561553 1567 raft_consensus.cc:2781] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:00:49.561676 1567 raft_consensus.cc:482] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:00:49.561672 1580 raft_consensus.cc:2781] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:00:49.561774 1567 raft_consensus.cc:3037] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:49.561800 1580 raft_consensus.cc:482] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:00:49.561888 1580 raft_consensus.cc:3037] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:49.564781 1580 raft_consensus.cc:504] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:49.564781 1567 raft_consensus.cc:504] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:49.565363 1567 leader_election.cc:288] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206 [CANDIDATE]: Term 1 election: Requested vote from peers 105aa6bd36964ed6b923cc2e4a9e31c9 (127.30.247.2:46397), 8463fe556af94c4ea931e4d060f9d7a5 (127.30.247.3:45275)
I20260925 05:00:49.565377 1580 leader_election.cc:288] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5 [CANDIDATE]: Term 1 election: Requested vote from peers 57227d8f465c4c4783262e70f0929206 (127.30.247.1:44329), 105aa6bd36964ed6b923cc2e4a9e31c9 (127.30.247.2:46397)
I20260925 05:00:49.565850 1302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd3fbc84a7044e4ab4afe2094081816" candidate_uuid: "57227d8f465c4c4783262e70f0929206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9"
I20260925 05:00:49.565892 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd3fbc84a7044e4ab4afe2094081816" candidate_uuid: "57227d8f465c4c4783262e70f0929206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8463fe556af94c4ea931e4d060f9d7a5"
I20260925 05:00:49.565908 1162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd3fbc84a7044e4ab4afe2094081816" candidate_uuid: "8463fe556af94c4ea931e4d060f9d7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57227d8f465c4c4783262e70f0929206"
I20260925 05:00:49.565941 1301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd3fbc84a7044e4ab4afe2094081816" candidate_uuid: "8463fe556af94c4ea931e4d060f9d7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9"
I20260925 05:00:49.566052 1302 raft_consensus.cc:3037] T 5dd3fbc84a7044e4ab4afe2094081816 P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:49.566138 1162 raft_consensus.cc:2379] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8463fe556af94c4ea931e4d060f9d7a5 in current term 1: Already voted for candidate 57227d8f465c4c4783262e70f0929206 in this term.
I20260925 05:00:49.566160 1442 raft_consensus.cc:2379] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 57227d8f465c4c4783262e70f0929206 in current term 1: Already voted for candidate 8463fe556af94c4ea931e4d060f9d7a5 in this term.
I20260925 05:00:49.566602 1368 leader_election.cc:302] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8463fe556af94c4ea931e4d060f9d7a5; no voters: 105aa6bd36964ed6b923cc2e4a9e31c9, 57227d8f465c4c4783262e70f0929206
I20260925 05:00:49.566876 1580 raft_consensus.cc:2726] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 05:00:49.568267 1302 raft_consensus.cc:2454] T 5dd3fbc84a7044e4ab4afe2094081816 P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57227d8f465c4c4783262e70f0929206 in term 1.
I20260925 05:00:49.568667 1090 leader_election.cc:302] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 105aa6bd36964ed6b923cc2e4a9e31c9, 57227d8f465c4c4783262e70f0929206; no voters: 8463fe556af94c4ea931e4d060f9d7a5
I20260925 05:00:49.568920 1567 raft_consensus.cc:2781] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:00:49.569061 1567 raft_consensus.cc:686] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206 [term 1 LEADER]: Becoming Leader. State: Replica: 57227d8f465c4c4783262e70f0929206, State: Running, Role: LEADER
I20260925 05:00:49.569337 1567 consensus_queue.cc:237] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:49.571810 995 catalog_manager.cc:5495] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57227d8f465c4c4783262e70f0929206 (127.30.247.1). New cstate: current_term: 1 leader_uuid: "57227d8f465c4c4783262e70f0929206" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } health_report { overall_health: UNKNOWN } } }
I20260925 05:00:49.613664 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:49.608475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57086 (local address 127.30.247.1:44329)
0925 05:00:49.609112 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:49.609116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:49.609132 (+ 16us) server_negotiation.cc:407] Connection header received
0925 05:00:49.609177 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:49.609180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:49.609235 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:49.609338 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:00:49.609731 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:49.610493 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:49.611774 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:49.612369 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:49.612501 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:49.612959 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:00:49.613051 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:00:49.613142 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:49.613289 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 05:00:49.613358 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":360,"threads_started":1}
I20260925 05:00:49.621047 1442 raft_consensus.cc:1261] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5 [term 1 FOLLOWER]: Refusing update from remote peer 57227d8f465c4c4783262e70f0929206: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:00:49.621338 1302 raft_consensus.cc:1261] T 5dd3fbc84a7044e4ab4afe2094081816 P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 1 FOLLOWER]: Refusing update from remote peer 57227d8f465c4c4783262e70f0929206: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:00:49.621568 1570 consensus_queue.cc:1035] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:00:49.621919 1567 consensus_queue.cc:1035] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206 [LEADER]: Connected to new peer: Peer: permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:00:49.659463 997 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:00:49.659690 997 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:00:49.662262 997 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e75816c69a345df9807f8da66ce85dd: Sending DeleteTablet for 3 replicas of tablet 5dd3fbc84a7044e4ab4afe2094081816
I20260925 05:00:49.662925 1136 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd3fbc84a7044e4ab4afe2094081816 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:00:49 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:57844
I20260925 05:00:49.662942 1277 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd3fbc84a7044e4ab4afe2094081816 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:00:49 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:48826
I20260925 05:00:49.663075 1417 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd3fbc84a7044e4ab4afe2094081816 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:00:49 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:38176
I20260925 05:00:49.663193 1588 tablet_replica.cc:330] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206: stopping tablet replica
I20260925 05:00:49.663326 1589 tablet_replica.cc:330] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5: stopping tablet replica
I20260925 05:00:49.663342 1590 tablet_replica.cc:330] T 5dd3fbc84a7044e4ab4afe2094081816 P 105aa6bd36964ed6b923cc2e4a9e31c9: stopping tablet replica
I20260925 05:00:49.663391 1588 raft_consensus.cc:2229] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:00:49.663537 1590 raft_consensus.cc:2229] T 5dd3fbc84a7044e4ab4afe2094081816 P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:00:49.663532 1589 raft_consensus.cc:2229] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:00:49.663666 1590 raft_consensus.cc:2258] T 5dd3fbc84a7044e4ab4afe2094081816 P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:00:49.663679 1588 raft_consensus.cc:2258] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:00:49.663697 1589 raft_consensus.cc:2258] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:00:49.664682 1588 ts_tablet_manager.cc:1905] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:00:49.664702 1590 ts_tablet_manager.cc:1905] T 5dd3fbc84a7044e4ab4afe2094081816 P 105aa6bd36964ed6b923cc2e4a9e31c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:00:49.665509 997 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:00:49.666119 997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:00:49.669981 1590 ts_tablet_manager.cc:1918] T 5dd3fbc84a7044e4ab4afe2094081816 P 105aa6bd36964ed6b923cc2e4a9e31c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:00:49.670125 1590 log.cc:1196] T 5dd3fbc84a7044e4ab4afe2094081816 P 105aa6bd36964ed6b923cc2e4a9e31c9: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-1/wal/wals/5dd3fbc84a7044e4ab4afe2094081816
I20260925 05:00:49.670536 1590 ts_tablet_manager.cc:1939] T 5dd3fbc84a7044e4ab4afe2094081816 P 105aa6bd36964ed6b923cc2e4a9e31c9: Deleting consensus metadata
I20260925 05:00:49.670557 1589 ts_tablet_manager.cc:1905] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:00:49.671502 985 catalog_manager.cc:4841] TS 105aa6bd36964ed6b923cc2e4a9e31c9 (127.30.247.2:46397): tablet 5dd3fbc84a7044e4ab4afe2094081816 (table test-table [id=57937c5059d7421da8a9c0dbc3ffbecc]) successfully deleted
I20260925 05:00:49.673192 1136 tablet_service.cc:1467] Processing CreateTablet for tablet d6fddb27ee554fb6ab2326a97d1966af (DEFAULT_TABLE table=test-table [id=c86ff64bd991404ea2511ac20b2907c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:00:49.673617 1136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6fddb27ee554fb6ab2326a97d1966af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:49.673615 1417 tablet_service.cc:1467] Processing CreateTablet for tablet d6fddb27ee554fb6ab2326a97d1966af (DEFAULT_TABLE table=test-table [id=c86ff64bd991404ea2511ac20b2907c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:00:49.673614 1588 ts_tablet_manager.cc:1918] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:00:49.673941 1588 log.cc:1196] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-0/wal/wals/5dd3fbc84a7044e4ab4afe2094081816
I20260925 05:00:49.674008 1417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6fddb27ee554fb6ab2326a97d1966af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:49.674487 1588 ts_tablet_manager.cc:1939] T 5dd3fbc84a7044e4ab4afe2094081816 P 57227d8f465c4c4783262e70f0929206: Deleting consensus metadata
I20260925 05:00:49.675470 982 catalog_manager.cc:4841] TS 57227d8f465c4c4783262e70f0929206 (127.30.247.1:44329): tablet 5dd3fbc84a7044e4ab4afe2094081816 (table test-table [id=57937c5059d7421da8a9c0dbc3ffbecc]) successfully deleted
I20260925 05:00:49.676944 1589 ts_tablet_manager.cc:1918] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:00:49.677101 1589 log.cc:1196] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-2/wal/wals/5dd3fbc84a7044e4ab4afe2094081816
I20260925 05:00:49.677500 1589 ts_tablet_manager.cc:1939] T 5dd3fbc84a7044e4ab4afe2094081816 P 8463fe556af94c4ea931e4d060f9d7a5: Deleting consensus metadata
I20260925 05:00:49.678503 983 catalog_manager.cc:4841] TS 8463fe556af94c4ea931e4d060f9d7a5 (127.30.247.3:45275): tablet 5dd3fbc84a7044e4ab4afe2094081816 (table test-table [id=57937c5059d7421da8a9c0dbc3ffbecc]) successfully deleted
I20260925 05:00:49.679920 1600 tablet_bootstrap.cc:492] T d6fddb27ee554fb6ab2326a97d1966af P 8463fe556af94c4ea931e4d060f9d7a5: Bootstrap starting.
I20260925 05:00:49.680001 1277 tablet_service.cc:1467] Processing CreateTablet for tablet d6fddb27ee554fb6ab2326a97d1966af (DEFAULT_TABLE table=test-table [id=c86ff64bd991404ea2511ac20b2907c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:00:49.680104 1601 tablet_bootstrap.cc:492] T d6fddb27ee554fb6ab2326a97d1966af P 57227d8f465c4c4783262e70f0929206: Bootstrap starting.
I20260925 05:00:49.680334 1277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6fddb27ee554fb6ab2326a97d1966af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:49.681510 1600 tablet_bootstrap.cc:654] T d6fddb27ee554fb6ab2326a97d1966af P 8463fe556af94c4ea931e4d060f9d7a5: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:49.681527 1601 tablet_bootstrap.cc:654] T d6fddb27ee554fb6ab2326a97d1966af P 57227d8f465c4c4783262e70f0929206: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:49.683876 1600 tablet_bootstrap.cc:492] T d6fddb27ee554fb6ab2326a97d1966af P 8463fe556af94c4ea931e4d060f9d7a5: No bootstrap required, opened a new log
I20260925 05:00:49.684033 1600 ts_tablet_manager.cc:1397] T d6fddb27ee554fb6ab2326a97d1966af P 8463fe556af94c4ea931e4d060f9d7a5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260925 05:00:49.684751 1600 raft_consensus.cc:348] T d6fddb27ee554fb6ab2326a97d1966af P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:49.684983 1600 raft_consensus.cc:374] T d6fddb27ee554fb6ab2326a97d1966af P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:49.685052 1600 raft_consensus.cc:729] T d6fddb27ee554fb6ab2326a97d1966af P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8463fe556af94c4ea931e4d060f9d7a5, State: Initialized, Role: FOLLOWER
I20260925 05:00:49.685292 1600 consensus_queue.cc:260] T d6fddb27ee554fb6ab2326a97d1966af P 8463fe556af94c4ea931e4d060f9d7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:49.685544 1599 tablet_bootstrap.cc:492] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9: Bootstrap starting.
I20260925 05:00:49.686874 1600 ts_tablet_manager.cc:1428] T d6fddb27ee554fb6ab2326a97d1966af P 8463fe556af94c4ea931e4d060f9d7a5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 05:00:49.687561 1599 tablet_bootstrap.cc:654] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:49.690006 1599 tablet_bootstrap.cc:492] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9: No bootstrap required, opened a new log
I20260925 05:00:49.690217 1599 ts_tablet_manager.cc:1397] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260925 05:00:49.691076 1599 raft_consensus.cc:348] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:49.691399 1599 raft_consensus.cc:374] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:49.691488 1599 raft_consensus.cc:729] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 105aa6bd36964ed6b923cc2e4a9e31c9, State: Initialized, Role: FOLLOWER
I20260925 05:00:49.691748 1599 consensus_queue.cc:260] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:49.692412 1599 ts_tablet_manager.cc:1428] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 05:00:49.693392 1601 tablet_bootstrap.cc:492] T d6fddb27ee554fb6ab2326a97d1966af P 57227d8f465c4c4783262e70f0929206: No bootstrap required, opened a new log
I20260925 05:00:49.693567 1601 ts_tablet_manager.cc:1397] T d6fddb27ee554fb6ab2326a97d1966af P 57227d8f465c4c4783262e70f0929206: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260925 05:00:49.694135 1601 raft_consensus.cc:348] T d6fddb27ee554fb6ab2326a97d1966af P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:49.694383 1601 raft_consensus.cc:374] T d6fddb27ee554fb6ab2326a97d1966af P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:49.694468 1601 raft_consensus.cc:729] T d6fddb27ee554fb6ab2326a97d1966af P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57227d8f465c4c4783262e70f0929206, State: Initialized, Role: FOLLOWER
I20260925 05:00:49.694660 1601 consensus_queue.cc:260] T d6fddb27ee554fb6ab2326a97d1966af P 57227d8f465c4c4783262e70f0929206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:49.695084 1601 ts_tablet_manager.cc:1428] T d6fddb27ee554fb6ab2326a97d1966af P 57227d8f465c4c4783262e70f0929206: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 05:00:49.701004 1605 raft_consensus.cc:482] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:00:49.701175 1605 raft_consensus.cc:504] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:49.702454 1605 leader_election.cc:288] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57227d8f465c4c4783262e70f0929206 (127.30.247.1:44329), 8463fe556af94c4ea931e4d060f9d7a5 (127.30.247.3:45275)
I20260925 05:00:49.711234 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:49.702903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:42791 (local address 127.30.247.1:44329)
0925 05:00:49.703066 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:49.703074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:49.703721 (+ 647us) server_negotiation.cc:407] Connection header received
0925 05:00:49.703941 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:49.703946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:49.704014 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:49.704111 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:00:49.704904 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:49.706004 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:49.708928 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:49.710146 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:49.710304 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:49.710373 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:49.710827 (+ 454us) server_negotiation.cc:299] Negotiation successful
0925 05:00:49.710966 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 05:00:49.711277 1617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:49.702795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:51431 (local address 127.30.247.3:45275)
0925 05:00:49.703317 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:49.703325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:49.703824 (+ 499us) server_negotiation.cc:407] Connection header received
0925 05:00:49.703963 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:49.703968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:49.704028 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:49.704118 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:00:49.704931 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:49.706025 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:49.708932 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:49.710147 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:49.710305 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:49.710373 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:49.710827 (+ 454us) server_negotiation.cc:299] Negotiation successful
0925 05:00:49.710965 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":288,"threads_started":1}
I20260925 05:00:49.711382 1618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:49.702729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:45275 (local address 127.30.247.2:51431)
0925 05:00:49.703737 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:49.703752 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:49.703855 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:49.704144 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:49.704148 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:49.704183 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:00:49.704752 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:49.704763 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:49.706195 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:49.706200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:49.708780 (+ 2580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:49.708792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:49.710422 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:49.710428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:49.710631 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:49.710682 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 05:00:49.710763 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 05:00:49.710924 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":228,"spinlock_wait_cycles":1408,"thread_start_us":767,"threads_started":1}
I20260925 05:00:49.711517 1616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:49.702684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:44329 (local address 127.30.247.2:42791)
0925 05:00:49.703601 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:49.703627 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:49.703775 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:49.704153 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:49.704157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:49.704189 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:00:49.704666 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:49.704681 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:49.706202 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:49.706205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:49.708777 (+ 2572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:49.708792 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:49.710410 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:49.710417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:49.710636 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:49.710667 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 05:00:49.710763 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 05:00:49.710927 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":239,"spinlock_wait_cycles":896,"thread_start_us":629,"threads_started":1}
I20260925 05:00:49.712019 1162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6fddb27ee554fb6ab2326a97d1966af" candidate_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57227d8f465c4c4783262e70f0929206" is_pre_election: true
I20260925 05:00:49.712021 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6fddb27ee554fb6ab2326a97d1966af" candidate_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8463fe556af94c4ea931e4d060f9d7a5" is_pre_election: true
I20260925 05:00:49.712245 1162 raft_consensus.cc:2454] T d6fddb27ee554fb6ab2326a97d1966af P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 105aa6bd36964ed6b923cc2e4a9e31c9 in term 0.
I20260925 05:00:49.712276 1442 raft_consensus.cc:2454] T d6fddb27ee554fb6ab2326a97d1966af P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 105aa6bd36964ed6b923cc2e4a9e31c9 in term 0.
I20260925 05:00:49.712702 1228 leader_election.cc:302] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 105aa6bd36964ed6b923cc2e4a9e31c9, 57227d8f465c4c4783262e70f0929206, 8463fe556af94c4ea931e4d060f9d7a5; no voters:
I20260925 05:00:49.713083 1605 raft_consensus.cc:2781] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:00:49.713212 1605 raft_consensus.cc:482] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:00:49.713315 1605 raft_consensus.cc:3037] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:49.715446 1605 raft_consensus.cc:504] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:49.716020 1605 leader_election.cc:288] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9 [CANDIDATE]: Term 1 election: Requested vote from peers 57227d8f465c4c4783262e70f0929206 (127.30.247.1:44329), 8463fe556af94c4ea931e4d060f9d7a5 (127.30.247.3:45275)
I20260925 05:00:49.716329 1162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6fddb27ee554fb6ab2326a97d1966af" candidate_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57227d8f465c4c4783262e70f0929206"
I20260925 05:00:49.716388 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6fddb27ee554fb6ab2326a97d1966af" candidate_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8463fe556af94c4ea931e4d060f9d7a5"
I20260925 05:00:49.716521 1162 raft_consensus.cc:3037] T d6fddb27ee554fb6ab2326a97d1966af P 57227d8f465c4c4783262e70f0929206 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:49.716602 1442 raft_consensus.cc:3037] T d6fddb27ee554fb6ab2326a97d1966af P 8463fe556af94c4ea931e4d060f9d7a5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:49.718415 1442 raft_consensus.cc:2454] T d6fddb27ee554fb6ab2326a97d1966af P 8463fe556af94c4ea931e4d060f9d7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 105aa6bd36964ed6b923cc2e4a9e31c9 in term 1.
I20260925 05:00:49.718417 1162 raft_consensus.cc:2454] T d6fddb27ee554fb6ab2326a97d1966af P 57227d8f465c4c4783262e70f0929206 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 105aa6bd36964ed6b923cc2e4a9e31c9 in term 1.
I20260925 05:00:49.718768 1228 leader_election.cc:302] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 105aa6bd36964ed6b923cc2e4a9e31c9, 57227d8f465c4c4783262e70f0929206; no voters:
I20260925 05:00:49.719019 1605 raft_consensus.cc:2781] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:00:49.719398 1605 raft_consensus.cc:686] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 1 LEADER]: Becoming Leader. State: Replica: 105aa6bd36964ed6b923cc2e4a9e31c9, State: Running, Role: LEADER
I20260925 05:00:49.719759 1605 consensus_queue.cc:237] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } }
I20260925 05:00:49.722851 997 catalog_manager.cc:5495] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 105aa6bd36964ed6b923cc2e4a9e31c9 (127.30.247.2). New cstate: current_term: 1 leader_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "105aa6bd36964ed6b923cc2e4a9e31c9" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 } health_report { overall_health: UNKNOWN } } }
I20260925 05:00:49.753212 1621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:49.747589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59924 (local address 127.30.247.2:46397)
0925 05:00:49.748091 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:49.748095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:49.748113 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:00:49.748183 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:49.748186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:49.748254 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:49.748386 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:00:49.748811 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:49.749621 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:49.750920 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:49.751348 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:49.751442 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:49.751887 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:00:49.752453 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:00:49.752658 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:49.752788 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 05:00:49.752848 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":301,"threads_started":1}
I20260925 05:00:49.766943 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:49.762019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:58585 (local address 127.30.247.1:44329)
0925 05:00:49.762233 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:49.762241 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:49.762726 (+ 485us) server_negotiation.cc:407] Connection header received
0925 05:00:49.762874 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:49.762878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:49.762938 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:49.763002 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:00:49.763427 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:49.764156 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:49.765473 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:49.765850 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:49.765947 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:49.766368 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:00:49.766461 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:00:49.766523 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:49.766685 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 05:00:49.766749 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:00:49.767019 1624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:49.761996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:44329 (local address 127.30.247.2:58585)
0925 05:00:49.762640 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:49.762655 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:49.762765 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:49.763024 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:49.763026 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:49.763035 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:00:49.763306 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:49.763313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:49.764273 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:49.764276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:49.765350 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:49.765362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:49.766014 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:49.766019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:49.766166 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:49.766215 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:00:49.766562 (+ 347us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:00:49.766573 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 05:00:49.766661 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 05:00:49.766758 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":324,"threads_started":1}
I20260925 05:00:49.774660 1442 raft_consensus.cc:1261] T d6fddb27ee554fb6ab2326a97d1966af P 8463fe556af94c4ea931e4d060f9d7a5 [term 1 FOLLOWER]: Refusing update from remote peer 105aa6bd36964ed6b923cc2e4a9e31c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:00:49.774643 1162 raft_consensus.cc:1261] T d6fddb27ee554fb6ab2326a97d1966af P 57227d8f465c4c4783262e70f0929206 [term 1 FOLLOWER]: Refusing update from remote peer 105aa6bd36964ed6b923cc2e4a9e31c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:00:49.775305 1605 consensus_queue.cc:1035] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57227d8f465c4c4783262e70f0929206" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 44329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:00:49.775856 1619 consensus_queue.cc:1035] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8463fe556af94c4ea931e4d060f9d7a5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:00:49.798673 1621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:49.793173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:47599 (local address 127.30.247.2:46397)
0925 05:00:49.793338 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:49.793345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:49.794013 (+ 668us) server_negotiation.cc:407] Connection header received
0925 05:00:49.794118 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:49.794122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:49.794176 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:49.794250 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:00:49.794592 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:49.795445 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:49.797078 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:49.797474 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:49.797608 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:49.797981 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:00:49.798081 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:00:49.798168 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:49.798296 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 05:00:49.798367 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 05:00:49.798779 1629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:49.793264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:46397 (local address 127.30.247.1:47599)
0925 05:00:49.793920 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:49.793938 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:49.794039 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:49.794256 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:49.794259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:49.794266 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:00:49.794485 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:49.794492 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:49.795553 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:49.795560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:49.796944 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:49.796955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:49.797637 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:49.797643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:49.797787 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:49.797819 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:00:49.798170 (+ 351us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:00:49.798184 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 05:00:49.798277 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 05:00:49.798417 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":361,"threads_started":1}
W20260925 05:00:49.799860 1572 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 05:00:51.388851 995 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:00:51.389099 995 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:00:51.392712 995 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e75816c69a345df9807f8da66ce85dd: Sending DeleteTablet for 3 replicas of tablet d6fddb27ee554fb6ab2326a97d1966af
I20260925 05:00:51.393368 1136 tablet_service.cc:1514] Processing DeleteTablet for tablet d6fddb27ee554fb6ab2326a97d1966af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:00:51 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:57844
I20260925 05:00:51.393460 1277 tablet_service.cc:1514] Processing DeleteTablet for tablet d6fddb27ee554fb6ab2326a97d1966af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:00:51 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:48826
I20260925 05:00:51.393827 1417 tablet_service.cc:1514] Processing DeleteTablet for tablet d6fddb27ee554fb6ab2326a97d1966af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:00:51 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:38176
I20260925 05:00:51.394220 1653 tablet_replica.cc:330] T d6fddb27ee554fb6ab2326a97d1966af P 57227d8f465c4c4783262e70f0929206: stopping tablet replica
I20260925 05:00:51.394462 1653 raft_consensus.cc:2229] T d6fddb27ee554fb6ab2326a97d1966af P 57227d8f465c4c4783262e70f0929206 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:00:51.394501 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 1070
I20260925 05:00:51.394731 1653 raft_consensus.cc:2258] T d6fddb27ee554fb6ab2326a97d1966af P 57227d8f465c4c4783262e70f0929206 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:00:51.395272 1655 tablet_replica.cc:330] T d6fddb27ee554fb6ab2326a97d1966af P 8463fe556af94c4ea931e4d060f9d7a5: stopping tablet replica
I20260925 05:00:51.395498 1655 raft_consensus.cc:2229] T d6fddb27ee554fb6ab2326a97d1966af P 8463fe556af94c4ea931e4d060f9d7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:00:51.395740 1655 raft_consensus.cc:2258] T d6fddb27ee554fb6ab2326a97d1966af P 8463fe556af94c4ea931e4d060f9d7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:00:51.396273 1654 tablet_replica.cc:330] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9: stopping tablet replica
I20260925 05:00:51.396507 1654 raft_consensus.cc:2229] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:00:51.396831 1653 ts_tablet_manager.cc:1905] T d6fddb27ee554fb6ab2326a97d1966af P 57227d8f465c4c4783262e70f0929206: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:00:51.397051 1654 raft_consensus.cc:2258] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:00:51.398105 1654 ts_tablet_manager.cc:1905] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:00:51.398510 1655 ts_tablet_manager.cc:1905] T d6fddb27ee554fb6ab2326a97d1966af P 8463fe556af94c4ea931e4d060f9d7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:00:51.403746 1655 ts_tablet_manager.cc:1918] T d6fddb27ee554fb6ab2326a97d1966af P 8463fe556af94c4ea931e4d060f9d7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 05:00:51.403877 1655 log.cc:1196] T d6fddb27ee554fb6ab2326a97d1966af P 8463fe556af94c4ea931e4d060f9d7a5: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-2/wal/wals/d6fddb27ee554fb6ab2326a97d1966af
I20260925 05:00:51.404313 1655 ts_tablet_manager.cc:1939] T d6fddb27ee554fb6ab2326a97d1966af P 8463fe556af94c4ea931e4d060f9d7a5: Deleting consensus metadata
I20260925 05:00:51.405308 983 catalog_manager.cc:4841] TS 8463fe556af94c4ea931e4d060f9d7a5 (127.30.247.3:45275): tablet d6fddb27ee554fb6ab2326a97d1966af (table test-table [id=c86ff64bd991404ea2511ac20b2907c9]) successfully deleted
I20260925 05:00:51.406030 1653 ts_tablet_manager.cc:1918] T d6fddb27ee554fb6ab2326a97d1966af P 57227d8f465c4c4783262e70f0929206: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 05:00:51.406092 1654 ts_tablet_manager.cc:1918] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 05:00:51.406160 1653 log.cc:1196] T d6fddb27ee554fb6ab2326a97d1966af P 57227d8f465c4c4783262e70f0929206: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-0/wal/wals/d6fddb27ee554fb6ab2326a97d1966af
I20260925 05:00:51.406239 1654 log.cc:1196] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312430481417-31708-0/minicluster-data/ts-1/wal/wals/d6fddb27ee554fb6ab2326a97d1966af
I20260925 05:00:51.406596 1653 ts_tablet_manager.cc:1939] T d6fddb27ee554fb6ab2326a97d1966af P 57227d8f465c4c4783262e70f0929206: Deleting consensus metadata
I20260925 05:00:51.406738 1654 ts_tablet_manager.cc:1939] T d6fddb27ee554fb6ab2326a97d1966af P 105aa6bd36964ed6b923cc2e4a9e31c9: Deleting consensus metadata
I20260925 05:00:51.407546 982 catalog_manager.cc:4841] TS 57227d8f465c4c4783262e70f0929206 (127.30.247.1:44329): tablet d6fddb27ee554fb6ab2326a97d1966af (table test-table [id=c86ff64bd991404ea2511ac20b2907c9]) successfully deleted
I20260925 05:00:51.407776 985 catalog_manager.cc:4841] TS 105aa6bd36964ed6b923cc2e4a9e31c9 (127.30.247.2:46397): tablet d6fddb27ee554fb6ab2326a97d1966af (table test-table [id=c86ff64bd991404ea2511ac20b2907c9]) successfully deleted
Sep 25 05:00:51 dist-test-slave-5n86 krb5kdc[953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312449, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.247.1@KRBTEST.COM
I20260925 05:00:51.423976 1657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:51.395759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57102 (local address 127.30.247.1:44329)
0925 05:00:51.407500 (+ 11741us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:51.407507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:51.408076 (+ 569us) server_negotiation.cc:407] Connection header received
0925 05:00:51.408120 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:51.408124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:51.408186 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:51.408270 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:51.409418 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:51.410102 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:51.411212 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:51.411591 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:51.411678 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:51.418609 (+ 6931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:51.418627 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:51.418644 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:51.418689 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:51.421550 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:51.422139 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:51.422146 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:51.422149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:51.422278 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:51.422590 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:51.422593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:51.422595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:51.423046 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:51.423285 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:51.423468 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 05:00:51.423628 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11614,"thread_start_us":11532,"threads_started":1}
W20260925 05:00:51.424688 1203 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57102
W20260925 05:00:51.425311 31708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 1070: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:00:51.425524 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 1070
I20260925 05:00:51.451038 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 1212
Sep 25 05:00:51 dist-test-slave-5n86 krb5kdc[953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312449, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.247.2@KRBTEST.COM
I20260925 05:00:51.465000 1658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:51.451738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59932 (local address 127.30.247.2:46397)
0925 05:00:51.452188 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:51.452193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:51.452221 (+ 28us) server_negotiation.cc:407] Connection header received
0925 05:00:51.452280 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:51.452284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:51.452337 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:51.452422 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:51.453640 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:51.454449 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:51.455574 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:51.455928 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:51.456015 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:51.459729 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:51.459744 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:51.459760 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:51.459792 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:51.462624 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:51.463147 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:51.463154 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:51.463156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:51.463285 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:51.463610 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:51.463613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:51.463615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:51.463951 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:51.464193 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:51.464436 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 05:00:51.464629 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":215,"threads_started":1}
W20260925 05:00:51.465574 1342 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59932
W20260925 05:00:51.466092 31708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 1212: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:00:51.466204 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 1212
I20260925 05:00:51.487473 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 1351
Sep 25 05:00:51 dist-test-slave-5n86 krb5kdc[953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312449, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.247.3@KRBTEST.COM
I20260925 05:00:51.502681 1659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:51.488121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42666 (local address 127.30.247.3:45275)
0925 05:00:51.488652 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:51.488657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:51.488673 (+ 16us) server_negotiation.cc:407] Connection header received
0925 05:00:51.488725 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:51.488729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:51.488787 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:51.488874 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:51.490138 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:51.490863 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:51.492049 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:51.492467 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:51.492701 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:51.496729 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:51.496744 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:51.496760 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:51.496794 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:51.500055 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:51.500739 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:51.500748 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:51.500750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:51.500875 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:51.501215 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:51.501221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:51.501224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:51.501564 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:51.501854 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:51.502142 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 05:00:51.502374 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":236,"threads_started":1}
W20260925 05:00:51.503345 1483 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42666
W20260925 05:00:51.503943 31708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 1351: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:00:51.504107 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 1351
I20260925 05:00:51.530953 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 966
I20260925 05:00:51.532289 1039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:00:51.657595 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 966
2026-09-25T05:00:51Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8006 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:00:51 dist-test-slave-5n86 krb5kdc[1663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:00:51 dist-test-slave-5n86 krb5kdc[1663](info): set up 2 sockets
Sep 25 05:00:51 dist-test-slave-5n86 krb5kdc[1663](info): commencing operation
krb5kdc: starting...
W20260925 05:00:52.737047 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.006s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:00:52 dist-test-slave-5n86 krb5kdc[1663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:00:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:00:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:00:52.915942 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:35495
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:44087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:35495
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:53.225858 1676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:53.226105 1676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:53.226281 1676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:53.233304 1676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:00:53.233412 1676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:53.233489 1676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:53.233578 1676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:00:53.233659 1676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:00:53.242965 1676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:44087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:35495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:35495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:53.244141 1676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:53.245415 1676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:53.256950 1681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:53.257025 1682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:53.257038 1684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:53.257634 1676 server_base.cc:1034] running on GCE node
I20260925 05:00:53.258272 1676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:53.259490 1676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:53.260681 1676 hybrid_clock.cc:648] HybridClock initialized: now 1790312453260636 us; error 29 us; skew 500 ppm
I20260925 05:00:53.261023 1676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:53 dist-test-slave-5n86 krb5kdc[1663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312453, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:53.263959 1676 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:00:53.265393 1676 webserver.cc:460] Webserver started at http://127.30.247.62:35285/ using document root <none> and password file <none>
I20260925 05:00:53.266012 1676 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:53.266106 1676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:53.266382 1676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:53.269124 1676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "3690a6da410543d1b0f0175c4caf9470"
format_stamp: "Formatted at 2026-09-25 05:00:53 on dist-test-slave-5n86"
server_key: "50a2d22069ce5fa44a4a3e85ccd34a9f"
server_key_iv: "2b27f7f6da8fea5e8d72488f865a47b7"
server_key_version: "encryptionkey@0"
I20260925 05:00:53.269783 1676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "3690a6da410543d1b0f0175c4caf9470"
format_stamp: "Formatted at 2026-09-25 05:00:53 on dist-test-slave-5n86"
server_key: "50a2d22069ce5fa44a4a3e85ccd34a9f"
server_key_iv: "2b27f7f6da8fea5e8d72488f865a47b7"
server_key_version: "encryptionkey@0"
I20260925 05:00:53.274459 1676 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 05:00:53.277614 1691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:53.278862 1676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:00:53.279013 1676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "3690a6da410543d1b0f0175c4caf9470"
format_stamp: "Formatted at 2026-09-25 05:00:53 on dist-test-slave-5n86"
server_key: "50a2d22069ce5fa44a4a3e85ccd34a9f"
server_key_iv: "2b27f7f6da8fea5e8d72488f865a47b7"
server_key_version: "encryptionkey@0"
I20260925 05:00:53.279204 1676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:53.298381 1676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:53.302476 1676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:53.302766 1676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:53.320708 1676 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:35495
I20260925 05:00:53.320712 1743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:35495 every 8 connection(s)
I20260925 05:00:53.322026 1676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:00:53.325639 1744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:53.330845 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 1676
I20260925 05:00:53.331065 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:00:53.331712 31708 external_mini_cluster.cc:1428] Setting key 7a88f80a43e4758e606014afe6f960b5
I20260925 05:00:53.336789 1744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3690a6da410543d1b0f0175c4caf9470: Bootstrap starting.
I20260925 05:00:53.340289 1744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3690a6da410543d1b0f0175c4caf9470: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:53.341439 1744 log.cc:824] T 00000000000000000000000000000000 P 3690a6da410543d1b0f0175c4caf9470: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:53.344650 1744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3690a6da410543d1b0f0175c4caf9470: No bootstrap required, opened a new log
Sep 25 05:00:53 dist-test-slave-5n86 krb5kdc[1663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:00:53.354321 1744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3690a6da410543d1b0f0175c4caf9470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3690a6da410543d1b0f0175c4caf9470" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35495 } }
I20260925 05:00:53.354745 1744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3690a6da410543d1b0f0175c4caf9470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:53.354851 1744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3690a6da410543d1b0f0175c4caf9470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3690a6da410543d1b0f0175c4caf9470, State: Initialized, Role: FOLLOWER
I20260925 05:00:53.355499 1744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3690a6da410543d1b0f0175c4caf9470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3690a6da410543d1b0f0175c4caf9470" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35495 } }
I20260925 05:00:53.355775 1744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3690a6da410543d1b0f0175c4caf9470 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:00:53.355913 1744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3690a6da410543d1b0f0175c4caf9470 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:00:53.356094 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:53.333726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60678 (local address 127.30.247.62:35495)
0925 05:00:53.334557 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:53.334573 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:53.334618 (+ 45us) server_negotiation.cc:407] Connection header received
0925 05:00:53.336908 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:53.336936 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:53.337400 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:53.337793 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:53.339042 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:53.341030 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:53.342641 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:53.343608 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:53.343795 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:53.347570 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:53.347581 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:53.347605 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:53.347657 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:53.351693 (+ 4036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:53.352347 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:53.352356 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:53.352364 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:53.352591 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:53.352983 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:53.352992 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:53.352996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:53.353673 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:53.353954 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:53.354338 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 05:00:53.354784 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":327,"threads_started":1}
I20260925 05:00:53.356097 1744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3690a6da410543d1b0f0175c4caf9470 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:53.358433 1744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3690a6da410543d1b0f0175c4caf9470 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3690a6da410543d1b0f0175c4caf9470" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35495 } }
I20260925 05:00:53.358950 1744 leader_election.cc:302] T 00000000000000000000000000000000 P 3690a6da410543d1b0f0175c4caf9470 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3690a6da410543d1b0f0175c4caf9470; no voters:
I20260925 05:00:53.359473 1744 leader_election.cc:288] T 00000000000000000000000000000000 P 3690a6da410543d1b0f0175c4caf9470 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:00:53.359620 1749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3690a6da410543d1b0f0175c4caf9470 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:00:53.360340 1749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3690a6da410543d1b0f0175c4caf9470 [term 1 LEADER]: Becoming Leader. State: Replica: 3690a6da410543d1b0f0175c4caf9470, State: Running, Role: LEADER
I20260925 05:00:53.360885 1749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3690a6da410543d1b0f0175c4caf9470 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3690a6da410543d1b0f0175c4caf9470" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35495 } }
I20260925 05:00:53.361542 1744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3690a6da410543d1b0f0175c4caf9470 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:00:53.363857 1751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3690a6da410543d1b0f0175c4caf9470 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3690a6da410543d1b0f0175c4caf9470. Latest consensus state: current_term: 1 leader_uuid: "3690a6da410543d1b0f0175c4caf9470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3690a6da410543d1b0f0175c4caf9470" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35495 } } }
I20260925 05:00:53.363766 1750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3690a6da410543d1b0f0175c4caf9470 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3690a6da410543d1b0f0175c4caf9470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3690a6da410543d1b0f0175c4caf9470" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35495 } } }
I20260925 05:00:53.364125 1751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3690a6da410543d1b0f0175c4caf9470 [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:53.364123 1750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3690a6da410543d1b0f0175c4caf9470 [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:53.365007 1756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:00:53.369866 1756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:00:53.379158 1756 catalog_manager.cc:1347] Generated new cluster ID: 13c5b00197f4450382d6b59308f6605a
I20260925 05:00:53.379277 1756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:00:53.390691 1756 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:00:53.391840 1756 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:00:53.400388 1756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3690a6da410543d1b0f0175c4caf9470: Generated new TSK 0
I20260925 05:00:53.401338 1756 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
I20260925 05:00:53.456494 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:0
--local_ip_for_outbound_sockets=127.30.247.1
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:35495
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:44087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:53.765503 1770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:53.765799 1770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:53.765946 1770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:53.773041 1770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:53.773135 1770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:53.773237 1770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:00:53.782261 1770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:44087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:35495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:53.783288 1770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:53.784557 1770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:53.796751 1775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:53.796787 1778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:53.796819 1776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:53.797435 1770 server_base.cc:1034] running on GCE node
I20260925 05:00:53.797928 1770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:53.798720 1770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:53.799932 1770 hybrid_clock.cc:648] HybridClock initialized: now 1790312453799864 us; error 50 us; skew 500 ppm
I20260925 05:00:53.800266 1770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:53 dist-test-slave-5n86 krb5kdc[1663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312453, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:53.803315 1770 init.cc:377] Logged in from keytab as kudu/127.30.247.1@KRBTEST.COM (short username kudu)
I20260925 05:00:53.804641 1770 webserver.cc:460] Webserver started at http://127.30.247.1:41039/ using document root <none> and password file <none>
I20260925 05:00:53.805238 1770 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:53.805331 1770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:53.805564 1770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:53.808337 1770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/data/instance:
uuid: "36980ead244f444195ec774094be60d4"
format_stamp: "Formatted at 2026-09-25 05:00:53 on dist-test-slave-5n86"
server_key: "af413cc3bb8987c40cb7eb4981ed1bce"
server_key_iv: "4f7c739e8043f8b138d3c660eabf01bd"
server_key_version: "encryptionkey@0"
I20260925 05:00:53.809146 1770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance:
uuid: "36980ead244f444195ec774094be60d4"
format_stamp: "Formatted at 2026-09-25 05:00:53 on dist-test-slave-5n86"
server_key: "af413cc3bb8987c40cb7eb4981ed1bce"
server_key_iv: "4f7c739e8043f8b138d3c660eabf01bd"
server_key_version: "encryptionkey@0"
I20260925 05:00:53.814004 1770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 05:00:53.817206 1785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:53.818420 1770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:00:53.818578 1770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "36980ead244f444195ec774094be60d4"
format_stamp: "Formatted at 2026-09-25 05:00:53 on dist-test-slave-5n86"
server_key: "af413cc3bb8987c40cb7eb4981ed1bce"
server_key_iv: "4f7c739e8043f8b138d3c660eabf01bd"
server_key_version: "encryptionkey@0"
I20260925 05:00:53.818760 1770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:53.834969 1770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:53.838730 1770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:53.838990 1770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:53.839881 1770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:00:53.841126 1770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:00:53.841205 1770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:53.841355 1770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:00:53.841418 1770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:53.873677 1770 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:35873
I20260925 05:00:53.873723 1897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:35873 every 8 connection(s)
I20260925 05:00:53.874948 1770 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:00:53.883055 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 1770
I20260925 05:00:53.883279 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance
I20260925 05:00:53.884034 31708 external_mini_cluster.cc:1428] Setting key 856b16e991a3adee269dc163abc731e4
Sep 25 05:00:53 dist-test-slave-5n86 krb5kdc[1663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312453, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:00:53.895427 1900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:53.877477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:58067 (local address 127.30.247.62:35495)
0925 05:00:53.877930 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:53.877934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:53.878777 (+ 843us) server_negotiation.cc:407] Connection header received
0925 05:00:53.880609 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:53.880615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:53.880704 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:53.880818 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:53.882360 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:53.883283 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:53.885082 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:53.885695 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:53.885842 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:53.889900 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:53.889905 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:53.889908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:53.889939 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:53.892495 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:53.893240 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:53.893247 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:53.893250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:53.893368 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:53.893779 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:53.893784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:53.893788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:53.894060 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:53.894214 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:53.894853 (+ 639us) server_negotiation.cc:299] Negotiation successful
0925 05:00:53.895056 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":248,"threads_started":1}
I20260925 05:00:53.896333 1901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:53.877897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:35495 (local address 127.30.247.1:58067)
0925 05:00:53.878597 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:53.878636 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:53.880320 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:53.881052 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:53.881060 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:53.881556 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:53.882205 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:53.882225 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:53.883465 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:53.883469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:53.884931 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:53.884945 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:53.885922 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:53.885927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:53.886215 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:53.886920 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:53.886957 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:53.889670 (+ 2713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:53.892672 (+ 3002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:53.892681 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:53.892697 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:53.893107 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:53.893472 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:53.893477 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:53.893481 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:53.893658 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:53.894187 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:53.894195 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:53.894516 (+ 321us) client_negotiation.cc:769] Sending connection context
0925 05:00:53.894787 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 05:00:53.895275 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":288,"threads_started":1}
I20260925 05:00:53.897838 1898 heartbeater.cc:347] Connected to a master server at 127.30.247.62:35495
I20260925 05:00:53.898154 1898 heartbeater.cc:464] Registering TS with master...
I20260925 05:00:53.899092 1898 heartbeater.cc:511] Master 127.30.247.62:35495 requested a full tablet report, sending...
I20260925 05:00:53.901360 1708 ts_manager.cc:194] Registered new tserver with Master: 36980ead244f444195ec774094be60d4 (127.30.247.1:35873)
WARNING: no policy specified for kudu/127.30.247.2@KRBTEST.COM; defaulting to no policy
I20260925 05:00:53.902926 1708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.1@KRBTEST.COM'} at 127.30.247.1:58067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
I20260925 05:00:53.924278 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:0
--local_ip_for_outbound_sockets=127.30.247.2
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:35495
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:44087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:54.235839 1904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:54.236078 1904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:54.236217 1904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:54.243345 1904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:54.243436 1904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:54.243630 1904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:00:54.253073 1904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:44087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:35495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:54.254253 1904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:54.255484 1904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:54.268148 1909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:54.268072 1910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:54.268126 1912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:54.268740 1904 server_base.cc:1034] running on GCE node
I20260925 05:00:54.269310 1904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:54.270083 1904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:54.271270 1904 hybrid_clock.cc:648] HybridClock initialized: now 1790312454271231 us; error 33 us; skew 500 ppm
I20260925 05:00:54.271508 1904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:54 dist-test-slave-5n86 krb5kdc[1663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312454, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:54.274443 1904 init.cc:377] Logged in from keytab as kudu/127.30.247.2@KRBTEST.COM (short username kudu)
I20260925 05:00:54.275841 1904 webserver.cc:460] Webserver started at http://127.30.247.2:37567/ using document root <none> and password file <none>
I20260925 05:00:54.276467 1904 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:54.276611 1904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:54.276921 1904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:54.279605 1904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/data/instance:
uuid: "00fdac5a8d0744ad95ac45abb7cb7f56"
format_stamp: "Formatted at 2026-09-25 05:00:54 on dist-test-slave-5n86"
server_key: "43f1186cae1cf9c449b426d18c9c43da"
server_key_iv: "cb871f2a70d9c0c18e7e40a79d3f41cb"
server_key_version: "encryptionkey@0"
I20260925 05:00:54.280357 1904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance:
uuid: "00fdac5a8d0744ad95ac45abb7cb7f56"
format_stamp: "Formatted at 2026-09-25 05:00:54 on dist-test-slave-5n86"
server_key: "43f1186cae1cf9c449b426d18c9c43da"
server_key_iv: "cb871f2a70d9c0c18e7e40a79d3f41cb"
server_key_version: "encryptionkey@0"
I20260925 05:00:54.285244 1904 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260925 05:00:54.288491 1919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:54.290012 1904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:00:54.290164 1904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "00fdac5a8d0744ad95ac45abb7cb7f56"
format_stamp: "Formatted at 2026-09-25 05:00:54 on dist-test-slave-5n86"
server_key: "43f1186cae1cf9c449b426d18c9c43da"
server_key_iv: "cb871f2a70d9c0c18e7e40a79d3f41cb"
server_key_version: "encryptionkey@0"
I20260925 05:00:54.290360 1904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:54.322631 1904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:54.326587 1904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:54.326857 1904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:54.327756 1904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:00:54.329072 1904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:00:54.329142 1904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:54.329299 1904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:00:54.329349 1904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:54.361385 1904 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:37115
I20260925 05:00:54.361442 2031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:37115 every 8 connection(s)
I20260925 05:00:54.362676 1904 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:00:54.369017 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 1904
I20260925 05:00:54.369240 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance
I20260925 05:00:54.369681 31708 external_mini_cluster.cc:1428] Setting key 69db32468436d3ee639e0cfba6b669f0
Sep 25 05:00:54 dist-test-slave-5n86 krb5kdc[1663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312454, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:00:54.381910 1900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:54.365100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:45765 (local address 127.30.247.62:35495)
0925 05:00:54.365286 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:54.365291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:54.366483 (+ 1192us) server_negotiation.cc:407] Connection header received
0925 05:00:54.368310 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:54.368315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:54.368402 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:54.368506 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:54.370061 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:54.370902 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:54.372508 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:54.372951 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:54.373079 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:54.376596 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:54.376601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:54.376604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:54.376634 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:54.379113 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:54.379788 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:54.379794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:54.379801 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:54.379926 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:54.380312 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:54.380319 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:54.380322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:54.380603 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:54.380756 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:54.381379 (+ 623us) server_negotiation.cc:299] Negotiation successful
0925 05:00:54.381586 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 05:00:54.383041 2034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:54.365610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:35495 (local address 127.30.247.2:45765)
0925 05:00:54.366302 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:54.366369 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:54.368055 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:54.368764 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:54.368772 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:54.369328 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:54.369911 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:54.369929 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:54.371048 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:54.371052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:54.372364 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:54.372375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:54.373171 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:54.373179 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:54.373387 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:54.374031 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:54.374056 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:54.376374 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:54.379248 (+ 2874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:54.379258 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:54.379274 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:54.379665 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:54.380028 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:54.380034 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:54.380038 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:54.380208 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:54.380718 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:54.380733 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:54.381038 (+ 305us) client_negotiation.cc:769] Sending connection context
0925 05:00:54.381293 (+ 255us) client_negotiation.cc:240] Negotiation successful
0925 05:00:54.381772 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":287,"threads_started":1}
I20260925 05:00:54.384403 2032 heartbeater.cc:347] Connected to a master server at 127.30.247.62:35495
I20260925 05:00:54.384714 2032 heartbeater.cc:464] Registering TS with master...
I20260925 05:00:54.385385 2032 heartbeater.cc:511] Master 127.30.247.62:35495 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.30.247.3@KRBTEST.COM; defaulting to no policy
I20260925 05:00:54.386704 1708 ts_manager.cc:194] Registered new tserver with Master: 00fdac5a8d0744ad95ac45abb7cb7f56 (127.30.247.2:37115)
I20260925 05:00:54.387516 1708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.2@KRBTEST.COM'} at 127.30.247.2:45765
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
I20260925 05:00:54.411502 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:0
--local_ip_for_outbound_sockets=127.30.247.3
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:35495
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:44087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:54.716513 2037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:54.716784 2037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:54.716936 2037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:54.723987 2037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:54.724100 2037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:54.724309 2037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:00:54.733656 2037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:44087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:35495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:54.734913 2037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:54.736179 2037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:54.748608 2042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:54.748626 2043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:54.748626 2045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:54.749203 2037 server_base.cc:1034] running on GCE node
I20260925 05:00:54.749818 2037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:54.750706 2037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:54.751905 2037 hybrid_clock.cc:648] HybridClock initialized: now 1790312454751867 us; error 35 us; skew 500 ppm
I20260925 05:00:54.752225 2037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:54 dist-test-slave-5n86 krb5kdc[1663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312454, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:54.755260 2037 init.cc:377] Logged in from keytab as kudu/127.30.247.3@KRBTEST.COM (short username kudu)
I20260925 05:00:54.756641 2037 webserver.cc:460] Webserver started at http://127.30.247.3:34553/ using document root <none> and password file <none>
I20260925 05:00:54.757277 2037 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:54.757409 2037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:54.757695 2037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:54.760387 2037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/data/instance:
uuid: "406b8e65a93e4588b15b118d69d73fd2"
format_stamp: "Formatted at 2026-09-25 05:00:54 on dist-test-slave-5n86"
server_key: "d649dcb7cc6f6348ab65e00826fc2d4e"
server_key_iv: "8e54013373d409228f56a3f7430d18a2"
server_key_version: "encryptionkey@0"
I20260925 05:00:54.761235 2037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance:
uuid: "406b8e65a93e4588b15b118d69d73fd2"
format_stamp: "Formatted at 2026-09-25 05:00:54 on dist-test-slave-5n86"
server_key: "d649dcb7cc6f6348ab65e00826fc2d4e"
server_key_iv: "8e54013373d409228f56a3f7430d18a2"
server_key_version: "encryptionkey@0"
I20260925 05:00:54.765956 2037 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 05:00:54.769222 2052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:54.770404 2037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260925 05:00:54.770556 2037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "406b8e65a93e4588b15b118d69d73fd2"
format_stamp: "Formatted at 2026-09-25 05:00:54 on dist-test-slave-5n86"
server_key: "d649dcb7cc6f6348ab65e00826fc2d4e"
server_key_iv: "8e54013373d409228f56a3f7430d18a2"
server_key_version: "encryptionkey@0"
I20260925 05:00:54.770740 2037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:54.786908 2037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:54.790570 2037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:54.790936 2037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:54.791805 2037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:00:54.793154 2037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:00:54.793247 2037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:54.793431 2037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:00:54.793483 2037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:54.825016 2037 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:38591
I20260925 05:00:54.825052 2164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:38591 every 8 connection(s)
I20260925 05:00:54.826351 2037 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:00:54.836138 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 2037
I20260925 05:00:54.836345 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance
I20260925 05:00:54.836812 31708 external_mini_cluster.cc:1428] Setting key fc63f69de6454962814fca220cd60764
Sep 25 05:00:54 dist-test-slave-5n86 krb5kdc[1663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312454, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:00:54.845755 1900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:54.828812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:36089 (local address 127.30.247.62:35495)
0925 05:00:54.828981 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:54.828986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:54.830275 (+ 1289us) server_negotiation.cc:407] Connection header received
0925 05:00:54.832124 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:54.832129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:54.832208 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:54.832297 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:54.833765 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:54.834492 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:54.836048 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:54.836600 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:54.836738 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:54.840366 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:54.840373 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:54.840377 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:54.840419 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:54.842942 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:54.843618 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:54.843624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:54.843627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:54.843744 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:54.844147 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:54.844153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:54.844156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:54.844411 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:54.844579 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:54.845219 (+ 640us) server_negotiation.cc:299] Negotiation successful
0925 05:00:54.845423 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:00:54.846650 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:54.829395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:35495 (local address 127.30.247.3:36089)
0925 05:00:54.830105 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:54.830152 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:54.831808 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:54.832495 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:54.832503 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:54.833062 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:54.833599 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:54.833628 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:54.834671 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:54.834675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:54.835895 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:54.835906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:54.836816 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:54.836825 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:54.837066 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:54.837776 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:54.837799 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:54.840177 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:54.843077 (+ 2900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:54.843085 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:54.843101 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:54.843499 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:54.843850 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:54.843854 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:54.843858 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:54.844034 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:54.844549 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:54.844558 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:54.844878 (+ 320us) client_negotiation.cc:769] Sending connection context
0925 05:00:54.845145 (+ 267us) client_negotiation.cc:240] Negotiation successful
0925 05:00:54.845636 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":283,"threads_started":1}
I20260925 05:00:54.847942 2165 heartbeater.cc:347] Connected to a master server at 127.30.247.62:35495
I20260925 05:00:54.848198 2165 heartbeater.cc:464] Registering TS with master...
I20260925 05:00:54.848937 2165 heartbeater.cc:511] Master 127.30.247.62:35495 requested a full tablet report, sending...
I20260925 05:00:54.850270 1708 ts_manager.cc:194] Registered new tserver with Master: 406b8e65a93e4588b15b118d69d73fd2 (127.30.247.3:38591)
I20260925 05:00:54.851328 1708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.3@KRBTEST.COM'} at 127.30.247.3:36089
I20260925 05:00:54.852119 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 05:00:54.882217 1900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:00:54.879790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60686 (local address 127.30.247.62:35495)
0925 05:00:54.879969 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:54.879974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:54.880448 (+ 474us) server_negotiation.cc:407] Connection header received
0925 05:00:54.880646 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:54.880650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:54.880738 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:54.880845 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:54.881956 (+ 1111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60686: BlockingRecv error: recv got EOF from 127.0.0.1:60686 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 05:00:54.885119 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 1770
W20260925 05:00:54.887755 31708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 1770: Not authorized: Client connection negotiation failed: client connection to 127.30.247.1:35873: server requires authentication, but client does not have Kerberos credentials available
I20260925 05:00:54.887938 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 1770
I20260925 05:00:54.903326 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 1904
W20260925 05:00:54.905890 31708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 1904: Not authorized: Client connection negotiation failed: client connection to 127.30.247.2:37115: server requires authentication, but client does not have Kerberos credentials available
I20260925 05:00:54.906028 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 1904
I20260925 05:00:54.920681 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 2037
W20260925 05:00:54.923230 31708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 2037: Not authorized: Client connection negotiation failed: client connection to 127.30.247.3:38591: server requires authentication, but client does not have Kerberos credentials available
I20260925 05:00:54.923381 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 2037
I20260925 05:00:54.938484 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 1676
I20260925 05:00:54.939677 1739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:00:55.040740 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 1676
2026-09-25T05:00:55Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3381 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:00:55 dist-test-slave-5n86 krb5kdc[2182](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:00:55 dist-test-slave-5n86 krb5kdc[2182](info): set up 2 sockets
Sep 25 05:00:55 dist-test-slave-5n86 krb5kdc[2182](info): commencing operation
krb5kdc: starting...
W20260925 05:00:56.120960 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:00:56 dist-test-slave-5n86 krb5kdc[2182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:00:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:00:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:00:56.294209 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:33823
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:45051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:33823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:56.599830 2195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:56.600144 2195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:56.600301 2195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:56.607285 2195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:00:56.607411 2195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:56.607510 2195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:56.607604 2195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:00:56.607687 2195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:00:56.616915 2195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:45051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:33823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:33823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:56.618063 2195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:56.619292 2195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:56.630213 2200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:56.630236 2201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:56.630538 2203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:56.631318 2195 server_base.cc:1034] running on GCE node
I20260925 05:00:56.632089 2195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:56.633397 2195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:56.634613 2195 hybrid_clock.cc:648] HybridClock initialized: now 1790312456634583 us; error 41 us; skew 500 ppm
I20260925 05:00:56.634946 2195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:56 dist-test-slave-5n86 krb5kdc[2182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312456, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:56.641541 2195 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:00:56.642982 2195 webserver.cc:460] Webserver started at http://127.30.247.62:42151/ using document root <none> and password file <none>
I20260925 05:00:56.643633 2195 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:56.643776 2195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:56.644071 2195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:56.646898 2195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "e9abff6c44444429a296462afa93956e"
format_stamp: "Formatted at 2026-09-25 05:00:56 on dist-test-slave-5n86"
server_key: "add0cfa7a1c92afd4f0f449a9b29c30a"
server_key_iv: "6b864251bfef4690848015a573f8d8d8"
server_key_version: "encryptionkey@0"
I20260925 05:00:56.647642 2195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "e9abff6c44444429a296462afa93956e"
format_stamp: "Formatted at 2026-09-25 05:00:56 on dist-test-slave-5n86"
server_key: "add0cfa7a1c92afd4f0f449a9b29c30a"
server_key_iv: "6b864251bfef4690848015a573f8d8d8"
server_key_version: "encryptionkey@0"
I20260925 05:00:56.652480 2195 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260925 05:00:56.655715 2210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:56.657058 2195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:00:56.657228 2195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "e9abff6c44444429a296462afa93956e"
format_stamp: "Formatted at 2026-09-25 05:00:56 on dist-test-slave-5n86"
server_key: "add0cfa7a1c92afd4f0f449a9b29c30a"
server_key_iv: "6b864251bfef4690848015a573f8d8d8"
server_key_version: "encryptionkey@0"
I20260925 05:00:56.657408 2195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:56.676472 2195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:56.680624 2195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:56.680893 2195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:56.698766 2195 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:33823
I20260925 05:00:56.698798 2261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:33823 every 8 connection(s)
I20260925 05:00:56.700007 2195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:00:56.703456 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:56.708863 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 2195
I20260925 05:00:56.709074 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:00:56.709720 31708 external_mini_cluster.cc:1428] Setting key 87fae58d8be300d765256eb0b103e920
I20260925 05:00:56.713476 2262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9abff6c44444429a296462afa93956e: Bootstrap starting.
I20260925 05:00:56.717048 2262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9abff6c44444429a296462afa93956e: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:56.718051 2262 log.cc:824] T 00000000000000000000000000000000 P e9abff6c44444429a296462afa93956e: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:56.720798 2262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9abff6c44444429a296462afa93956e: No bootstrap required, opened a new log
Sep 25 05:00:56 dist-test-slave-5n86 krb5kdc[2182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:00:56.728742 2262 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9abff6c44444429a296462afa93956e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9abff6c44444429a296462afa93956e" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 33823 } }
I20260925 05:00:56.729063 2262 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9abff6c44444429a296462afa93956e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:56.729168 2262 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9abff6c44444429a296462afa93956e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9abff6c44444429a296462afa93956e, State: Initialized, Role: FOLLOWER
I20260925 05:00:56.729769 2262 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9abff6c44444429a296462afa93956e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9abff6c44444429a296462afa93956e" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 33823 } }
I20260925 05:00:56.729993 2262 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9abff6c44444429a296462afa93956e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:00:56.730129 2262 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9abff6c44444429a296462afa93956e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:00:56.730302 2262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9abff6c44444429a296462afa93956e [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:56.731798 2265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:56.711443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58124 (local address 127.30.247.62:33823)
0925 05:00:56.712204 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:56.712221 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:56.712259 (+ 38us) server_negotiation.cc:407] Connection header received
0925 05:00:56.713963 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:56.713984 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:56.714344 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:56.714639 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:56.716600 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:56.717908 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:56.719447 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:56.720052 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:56.720179 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:56.724790 (+ 4611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:56.724798 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:56.724811 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:56.724851 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:56.727850 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:56.728497 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:56.728510 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:56.728515 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:56.728738 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:56.729103 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:56.729108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:56.729116 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:56.729657 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:56.729944 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:56.730425 (+ 481us) server_negotiation.cc:299] Negotiation successful
0925 05:00:56.730815 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":346,"threads_started":1}
I20260925 05:00:56.732939 2262 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9abff6c44444429a296462afa93956e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9abff6c44444429a296462afa93956e" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 33823 } }
I20260925 05:00:56.733414 2262 leader_election.cc:302] T 00000000000000000000000000000000 P e9abff6c44444429a296462afa93956e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9abff6c44444429a296462afa93956e; no voters:
I20260925 05:00:56.733989 2262 leader_election.cc:288] T 00000000000000000000000000000000 P e9abff6c44444429a296462afa93956e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:00:56.734135 2267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9abff6c44444429a296462afa93956e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:00:56.735912 2267 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9abff6c44444429a296462afa93956e [term 1 LEADER]: Becoming Leader. State: Replica: e9abff6c44444429a296462afa93956e, State: Running, Role: LEADER
I20260925 05:00:56.736155 31708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:00:56.736459 2267 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9abff6c44444429a296462afa93956e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9abff6c44444429a296462afa93956e" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 33823 } }
I20260925 05:00:56.736886 2262 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9abff6c44444429a296462afa93956e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:00:56.738893 2269 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9abff6c44444429a296462afa93956e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9abff6c44444429a296462afa93956e. Latest consensus state: current_term: 1 leader_uuid: "e9abff6c44444429a296462afa93956e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9abff6c44444429a296462afa93956e" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 33823 } } }
I20260925 05:00:56.739219 2269 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9abff6c44444429a296462afa93956e [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:56.739730 2268 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9abff6c44444429a296462afa93956e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9abff6c44444429a296462afa93956e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9abff6c44444429a296462afa93956e" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 33823 } } }
I20260925 05:00:56.740064 2268 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9abff6c44444429a296462afa93956e [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:56.743697 2272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:00:56.750015 2272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:00:56.763283 2272 catalog_manager.cc:1347] Generated new cluster ID: c1556ec559214c82a36653b5f3666e40
I20260925 05:00:56.763387 2272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:00:56.785297 2265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:56.776351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58132 (local address 127.30.247.62:33823)
0925 05:00:56.776535 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:56.776542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:56.777047 (+ 505us) server_negotiation.cc:407] Connection header received
0925 05:00:56.777240 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:56.777246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:56.777321 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:56.777429 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:56.778665 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:56.779719 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:56.781597 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:56.782246 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:56.782350 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:56.783535 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:56.783541 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:56.783546 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:00:56.784359 (+ 813us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:56.784422 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:00:56.784455 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:56.784576 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:56.784728 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 05:00:56.784817 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 05:00:56.797701 2272 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:00:56.799482 2272 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:00:56.804399 2272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9abff6c44444429a296462afa93956e: Generated new TSK 0
I20260925 05:00:56.805248 2272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:00:56.816176 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 2195
I20260925 05:00:56.817674 2257 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:00:56.907343 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 2195
2026-09-25T05:00:56Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1861 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:00:56 dist-test-slave-5n86 krb5kdc[2297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:00:56 dist-test-slave-5n86 krb5kdc[2297](info): set up 2 sockets
Sep 25 05:00:56 dist-test-slave-5n86 krb5kdc[2297](info): commencing operation
krb5kdc: starting...
W20260925 05:00:57.985011 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:00:58 dist-test-slave-5n86 krb5kdc[2297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:00:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:00:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:00:58.164096 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:37731
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:42685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:37731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:58.465819 2310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:58.466079 2310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:58.466234 2310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:58.473156 2310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:00:58.473246 2310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:58.473357 2310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:58.473454 2310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:00:58.473541 2310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:00:58.482627 2310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:42685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:37731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:37731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:58.483816 2310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:58.485038 2310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:58.496109 2316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:58.496224 2315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:58.496176 2318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:58.497614 2310 server_base.cc:1034] running on GCE node
I20260925 05:00:58.498469 2310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:58.499753 2310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:58.500960 2310 hybrid_clock.cc:648] HybridClock initialized: now 1790312458500916 us; error 31 us; skew 500 ppm
I20260925 05:00:58.501258 2310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:58 dist-test-slave-5n86 krb5kdc[2297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312458, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:58.504204 2310 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:00:58.505646 2310 webserver.cc:460] Webserver started at http://127.30.247.62:41325/ using document root <none> and password file <none>
I20260925 05:00:58.506333 2310 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:58.506469 2310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:58.506790 2310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:58.509821 2310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "df958243d76345829a0474069df4cd2b"
format_stamp: "Formatted at 2026-09-25 05:00:58 on dist-test-slave-5n86"
server_key: "9599978ae769339470ca9c0b82c9e6ed"
server_key_iv: "eab0c133226c6e23fb9fde750b588933"
server_key_version: "encryptionkey@0"
I20260925 05:00:58.510584 2310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "df958243d76345829a0474069df4cd2b"
format_stamp: "Formatted at 2026-09-25 05:00:58 on dist-test-slave-5n86"
server_key: "9599978ae769339470ca9c0b82c9e6ed"
server_key_iv: "eab0c133226c6e23fb9fde750b588933"
server_key_version: "encryptionkey@0"
I20260925 05:00:58.515689 2310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 05:00:58.519019 2325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:58.520347 2310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:00:58.520571 2310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "df958243d76345829a0474069df4cd2b"
format_stamp: "Formatted at 2026-09-25 05:00:58 on dist-test-slave-5n86"
server_key: "9599978ae769339470ca9c0b82c9e6ed"
server_key_iv: "eab0c133226c6e23fb9fde750b588933"
server_key_version: "encryptionkey@0"
I20260925 05:00:58.520798 2310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:58.535015 2310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:58.538950 2310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:58.539244 2310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:58.557880 2310 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:37731
I20260925 05:00:58.557912 2376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:37731 every 8 connection(s)
I20260925 05:00:58.559223 2310 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:00:58.562722 2377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:58.569281 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 2310
I20260925 05:00:58.569499 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:00:58.570109 31708 external_mini_cluster.cc:1428] Setting key bfb3bda0cd4319be5ae0b621a8e3ccc7
I20260925 05:00:58.572606 2377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df958243d76345829a0474069df4cd2b: Bootstrap starting.
I20260925 05:00:58.575640 2377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df958243d76345829a0474069df4cd2b: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:58.576912 2377 log.cc:824] T 00000000000000000000000000000000 P df958243d76345829a0474069df4cd2b: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:58.579854 2377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df958243d76345829a0474069df4cd2b: No bootstrap required, opened a new log
Sep 25 05:00:58 dist-test-slave-5n86 krb5kdc[2297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:00:58.588707 2377 raft_consensus.cc:348] T 00000000000000000000000000000000 P df958243d76345829a0474069df4cd2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df958243d76345829a0474069df4cd2b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37731 } }
I20260925 05:00:58.589044 2377 raft_consensus.cc:374] T 00000000000000000000000000000000 P df958243d76345829a0474069df4cd2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:58.589139 2377 raft_consensus.cc:729] T 00000000000000000000000000000000 P df958243d76345829a0474069df4cd2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df958243d76345829a0474069df4cd2b, State: Initialized, Role: FOLLOWER
I20260925 05:00:58.589764 2377 consensus_queue.cc:260] T 00000000000000000000000000000000 P df958243d76345829a0474069df4cd2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df958243d76345829a0474069df4cd2b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37731 } }
I20260925 05:00:58.589978 2377 raft_consensus.cc:388] T 00000000000000000000000000000000 P df958243d76345829a0474069df4cd2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:00:58.590131 2377 raft_consensus.cc:482] T 00000000000000000000000000000000 P df958243d76345829a0474069df4cd2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:00:58.590323 2377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df958243d76345829a0474069df4cd2b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:58.593150 2380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:58.572001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44400 (local address 127.30.247.62:37731)
0925 05:00:58.572840 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:58.572851 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:58.572890 (+ 39us) server_negotiation.cc:407] Connection header received
0925 05:00:58.574532 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:58.574552 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:58.574903 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:58.575190 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:58.576930 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:58.578610 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:58.580343 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:58.581042 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:58.581169 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:58.585616 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:58.585623 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:58.585636 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:58.585671 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:58.588700 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:58.589525 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:58.589532 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:58.589538 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:58.589684 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:58.590151 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:58.590156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:58.590160 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:58.590848 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:58.591164 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:58.591452 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 05:00:58.591870 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":366,"threads_started":1}
I20260925 05:00:58.593490 2377 raft_consensus.cc:504] T 00000000000000000000000000000000 P df958243d76345829a0474069df4cd2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df958243d76345829a0474069df4cd2b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37731 } }
I20260925 05:00:58.593963 2377 leader_election.cc:302] T 00000000000000000000000000000000 P df958243d76345829a0474069df4cd2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df958243d76345829a0474069df4cd2b; no voters:
I20260925 05:00:58.594589 2377 leader_election.cc:288] T 00000000000000000000000000000000 P df958243d76345829a0474069df4cd2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:00:58.594748 2382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df958243d76345829a0474069df4cd2b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:00:58.596309 2382 raft_consensus.cc:686] T 00000000000000000000000000000000 P df958243d76345829a0474069df4cd2b [term 1 LEADER]: Becoming Leader. State: Replica: df958243d76345829a0474069df4cd2b, State: Running, Role: LEADER
I20260925 05:00:58.596684 2377 sys_catalog.cc:564] T 00000000000000000000000000000000 P df958243d76345829a0474069df4cd2b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:00:58.596901 2382 consensus_queue.cc:237] T 00000000000000000000000000000000 P df958243d76345829a0474069df4cd2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df958243d76345829a0474069df4cd2b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37731 } }
I20260925 05:00:58.602872 2384 sys_catalog.cc:455] T 00000000000000000000000000000000 P df958243d76345829a0474069df4cd2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader df958243d76345829a0474069df4cd2b. Latest consensus state: current_term: 1 leader_uuid: "df958243d76345829a0474069df4cd2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df958243d76345829a0474069df4cd2b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37731 } } }
I20260925 05:00:58.603197 2384 sys_catalog.cc:458] T 00000000000000000000000000000000 P df958243d76345829a0474069df4cd2b [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:58.603965 2383 sys_catalog.cc:455] T 00000000000000000000000000000000 P df958243d76345829a0474069df4cd2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df958243d76345829a0474069df4cd2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df958243d76345829a0474069df4cd2b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37731 } } }
I20260925 05:00:58.604403 2383 sys_catalog.cc:458] T 00000000000000000000000000000000 P df958243d76345829a0474069df4cd2b [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:58.605799 2393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:00:58.610647 2393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:00:58.618590 2393 catalog_manager.cc:1347] Generated new cluster ID: 7571e035eef146a8bf1d6eefd68650ee
I20260925 05:00:58.618695 2393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:00:58.627774 2393 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:00:58.629024 2393 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:00:58.639497 2393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df958243d76345829a0474069df4cd2b: Generated new TSK 0
I20260925 05:00:58.640295 2393 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
I20260925 05:00:58.687928 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:0
--local_ip_for_outbound_sockets=127.30.247.1
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:37731
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:42685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:59.007879 2403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:59.008131 2403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:59.008221 2403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:59.015228 2403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:59.015321 2403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:59.015424 2403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:00:59.024528 2403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:42685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:37731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:59.025774 2403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:59.026968 2403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:59.039551 2411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:59.039623 2408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:59.039599 2409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:59.040230 2403 server_base.cc:1034] running on GCE node
I20260925 05:00:59.040812 2403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:59.041616 2403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:59.042811 2403 hybrid_clock.cc:648] HybridClock initialized: now 1790312459042762 us; error 40 us; skew 500 ppm
I20260925 05:00:59.043129 2403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:59 dist-test-slave-5n86 krb5kdc[2297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312459, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:59.046286 2403 init.cc:377] Logged in from keytab as kudu/127.30.247.1@KRBTEST.COM (short username kudu)
I20260925 05:00:59.047576 2403 webserver.cc:460] Webserver started at http://127.30.247.1:40179/ using document root <none> and password file <none>
I20260925 05:00:59.048209 2403 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:59.048336 2403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:59.048691 2403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:59.051384 2403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/data/instance:
uuid: "7046ff1286d54a54bbcc2ff02ae181c0"
format_stamp: "Formatted at 2026-09-25 05:00:59 on dist-test-slave-5n86"
server_key: "19cef7a4eece5de72a4948c04ba4b8cb"
server_key_iv: "304e855c495e6c3cbd52eea09c61ff5a"
server_key_version: "encryptionkey@0"
I20260925 05:00:59.052122 2403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance:
uuid: "7046ff1286d54a54bbcc2ff02ae181c0"
format_stamp: "Formatted at 2026-09-25 05:00:59 on dist-test-slave-5n86"
server_key: "19cef7a4eece5de72a4948c04ba4b8cb"
server_key_iv: "304e855c495e6c3cbd52eea09c61ff5a"
server_key_version: "encryptionkey@0"
I20260925 05:00:59.056962 2403 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260925 05:00:59.060087 2418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:59.061434 2403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:00:59.061599 2403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "7046ff1286d54a54bbcc2ff02ae181c0"
format_stamp: "Formatted at 2026-09-25 05:00:59 on dist-test-slave-5n86"
server_key: "19cef7a4eece5de72a4948c04ba4b8cb"
server_key_iv: "304e855c495e6c3cbd52eea09c61ff5a"
server_key_version: "encryptionkey@0"
I20260925 05:00:59.061787 2403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:59.090552 2403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:59.094851 2403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:59.095182 2403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:59.096114 2403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:00:59.097592 2403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:00:59.097702 2403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:59.097803 2403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:00:59.097856 2403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:59.131853 2403 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:42567
I20260925 05:00:59.131950 2531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:42567 every 8 connection(s)
I20260925 05:00:59.133201 2403 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:00:59.136137 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 2403
I20260925 05:00:59.136343 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance
I20260925 05:00:59.137043 31708 external_mini_cluster.cc:1428] Setting key 33e4dd8ec4e477cd006362ea618e92e1
Sep 25 05:00:59 dist-test-slave-5n86 krb5kdc[2297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312459, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.247.2@KRBTEST.COM; defaulting to no policy
I20260925 05:00:59.155445 2534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:59.135877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:50353 (local address 127.30.247.62:37731)
0925 05:00:59.136359 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:59.136366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:59.137571 (+ 1205us) server_negotiation.cc:407] Connection header received
0925 05:00:59.139919 (+ 2348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:59.139927 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:59.140008 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:59.140126 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:59.142106 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:59.143248 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:59.145388 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:59.145810 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:59.145955 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:59.149525 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:59.149531 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:59.149534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:59.149567 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:59.152188 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:59.153038 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:59.153043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:59.153046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:59.153164 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:59.153635 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:59.153639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:59.153641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:59.153909 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:59.154073 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:59.154824 (+ 751us) server_negotiation.cc:299] Negotiation successful
0925 05:00:59.155049 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":232,"threads_started":1}
I20260925 05:00:59.156500 2535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:59.136481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37731 (local address 127.30.247.1:50353)
0925 05:00:59.137395 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:59.137442 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:59.139585 (+ 2143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:59.140333 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:59.140342 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:59.141110 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:59.141921 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:59.141946 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:59.143400 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:59.143407 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:59.145218 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:59.145230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:59.146054 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:59.146062 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:59.146318 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:59.147011 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:59.147034 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:59.149294 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:59.152349 (+ 3055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:59.152358 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:59.152384 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:59.152899 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:59.153288 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:59.153294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:59.153298 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:59.153500 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:59.154043 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:59.154051 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:59.154444 (+ 393us) client_negotiation.cc:769] Sending connection context
0925 05:00:59.154719 (+ 275us) client_negotiation.cc:240] Negotiation successful
0925 05:00:59.155299 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":364,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.2@KRBTEST.COM" created.
I20260925 05:00:59.158169 2532 heartbeater.cc:347] Connected to a master server at 127.30.247.62:37731
I20260925 05:00:59.158457 2532 heartbeater.cc:464] Registering TS with master...
I20260925 05:00:59.159153 2532 heartbeater.cc:511] Master 127.30.247.62:37731 requested a full tablet report, sending...
I20260925 05:00:59.161177 2342 ts_manager.cc:194] Registered new tserver with Master: 7046ff1286d54a54bbcc2ff02ae181c0 (127.30.247.1:42567)
I20260925 05:00:59.162559 2342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.1@KRBTEST.COM'} at 127.30.247.1:50353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
I20260925 05:00:59.180003 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:0
--local_ip_for_outbound_sockets=127.30.247.2
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:37731
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:42685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:59.484885 2538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:59.485134 2538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:59.485278 2538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:59.492182 2538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:59.492269 2538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:59.492467 2538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:00:59.501688 2538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:42685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:37731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:59.502816 2538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:59.504009 2538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:59.516371 2544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:59.516419 2546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:59.516449 2543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:59.517127 2538 server_base.cc:1034] running on GCE node
I20260925 05:00:59.517693 2538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:59.518672 2538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:59.519881 2538 hybrid_clock.cc:648] HybridClock initialized: now 1790312459519827 us; error 52 us; skew 500 ppm
I20260925 05:00:59.520165 2538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:59 dist-test-slave-5n86 krb5kdc[2297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312459, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:59.523078 2538 init.cc:377] Logged in from keytab as kudu/127.30.247.2@KRBTEST.COM (short username kudu)
I20260925 05:00:59.524520 2538 webserver.cc:460] Webserver started at http://127.30.247.2:32971/ using document root <none> and password file <none>
I20260925 05:00:59.525246 2538 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:59.525394 2538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:59.525691 2538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:59.528836 2538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/data/instance:
uuid: "139738dfb30544dc96eeb7b9db3c02f7"
format_stamp: "Formatted at 2026-09-25 05:00:59 on dist-test-slave-5n86"
server_key: "421d99b2e1e5716308afe955f0ea7619"
server_key_iv: "885e4f3129fb5637ea0587975dfde926"
server_key_version: "encryptionkey@0"
I20260925 05:00:59.529582 2538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance:
uuid: "139738dfb30544dc96eeb7b9db3c02f7"
format_stamp: "Formatted at 2026-09-25 05:00:59 on dist-test-slave-5n86"
server_key: "421d99b2e1e5716308afe955f0ea7619"
server_key_iv: "885e4f3129fb5637ea0587975dfde926"
server_key_version: "encryptionkey@0"
I20260925 05:00:59.534392 2538 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 05:00:59.537679 2553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:59.539001 2538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:00:59.539193 2538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "139738dfb30544dc96eeb7b9db3c02f7"
format_stamp: "Formatted at 2026-09-25 05:00:59 on dist-test-slave-5n86"
server_key: "421d99b2e1e5716308afe955f0ea7619"
server_key_iv: "885e4f3129fb5637ea0587975dfde926"
server_key_version: "encryptionkey@0"
I20260925 05:00:59.539397 2538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:59.553575 2538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:59.557324 2538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:59.557628 2538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:59.558521 2538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:00:59.559834 2538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:00:59.559935 2538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:59.560073 2538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:00:59.560146 2538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:59.596103 2538 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:34851
I20260925 05:00:59.596149 2665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:34851 every 8 connection(s)
I20260925 05:00:59.597460 2538 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:00:59.605156 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 2538
I20260925 05:00:59.605326 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance
I20260925 05:00:59.605796 31708 external_mini_cluster.cc:1428] Setting key 6837b398cbcf5b492285c37fdac05c33
Sep 25 05:00:59 dist-test-slave-5n86 krb5kdc[2297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312459, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:00:59.616602 2534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:59.599982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:38535 (local address 127.30.247.62:37731)
0925 05:00:59.600162 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:59.600166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:59.601247 (+ 1081us) server_negotiation.cc:407] Connection header received
0925 05:00:59.603005 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:59.603009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:59.603077 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:59.603164 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:59.604569 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:59.605353 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:59.607096 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:59.607598 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:59.607722 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:59.611197 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:59.611202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:59.611205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:59.611235 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:59.613770 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:59.614455 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:59.614468 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:59.614470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:59.614585 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:59.614994 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:59.615000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:59.615003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:59.615249 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:59.615418 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:59.616024 (+ 606us) server_negotiation.cc:299] Negotiation successful
0925 05:00:59.616232 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 05:00:59.617499 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:59.600344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37731 (local address 127.30.247.2:38535)
0925 05:00:59.601076 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:59.601112 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:59.602758 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:59.603381 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:59.603388 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:59.603845 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:59.604387 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:59.604403 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:59.605525 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:59.605529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:59.606951 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:59.606963 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:59.607815 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:59.607822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:59.608040 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:59.608729 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:59.608749 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:59.610982 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:59.613910 (+ 2928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:59.613919 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:59.613933 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:59.614331 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:59.614699 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:59.614704 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:59.614707 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:59.614885 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:59.615374 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:59.615381 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:59.615692 (+ 311us) client_negotiation.cc:769] Sending connection context
0925 05:00:59.615935 (+ 243us) client_negotiation.cc:240] Negotiation successful
0925 05:00:59.616424 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":333,"threads_started":1}
I20260925 05:00:59.618932 2666 heartbeater.cc:347] Connected to a master server at 127.30.247.62:37731
I20260925 05:00:59.619225 2666 heartbeater.cc:464] Registering TS with master...
I20260925 05:00:59.619920 2666 heartbeater.cc:511] Master 127.30.247.62:37731 requested a full tablet report, sending...
I20260925 05:00:59.621356 2342 ts_manager.cc:194] Registered new tserver with Master: 139738dfb30544dc96eeb7b9db3c02f7 (127.30.247.2:34851)
I20260925 05:00:59.622542 2342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.2@KRBTEST.COM'} at 127.30.247.2:38535
WARNING: no policy specified for kudu/127.30.247.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
I20260925 05:00:59.647437 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:0
--local_ip_for_outbound_sockets=127.30.247.3
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:37731
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:42685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:59.953872 2671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:59.954141 2671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:59.954233 2671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:59.961365 2671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:59.961499 2671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:59.961666 2671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:00:59.973562 2671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:42685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:37731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:59.974829 2671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:59.976105 2671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:59.987682 2676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:59.988485 2677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:59.989950 2679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:59.990353 2671 server_base.cc:1034] running on GCE node
I20260925 05:00:59.990911 2671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:59.991779 2671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:59.993010 2671 hybrid_clock.cc:648] HybridClock initialized: now 1790312459992950 us; error 48 us; skew 500 ppm
I20260925 05:00:59.993315 2671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:59 dist-test-slave-5n86 krb5kdc[2297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312459, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:59.996264 2671 init.cc:377] Logged in from keytab as kudu/127.30.247.3@KRBTEST.COM (short username kudu)
I20260925 05:00:59.997892 2671 webserver.cc:460] Webserver started at http://127.30.247.3:40089/ using document root <none> and password file <none>
I20260925 05:00:59.998554 2671 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:59.998701 2671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:59.998988 2671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:00.001824 2671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/data/instance:
uuid: "9060190edd194c338535a79618928568"
format_stamp: "Formatted at 2026-09-25 05:00:59 on dist-test-slave-5n86"
server_key: "9d88f1a5886654abcdbac1a94e101445"
server_key_iv: "ed58c98d7d61b3fa9b39247f864e6f03"
server_key_version: "encryptionkey@0"
I20260925 05:01:00.002691 2671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance:
uuid: "9060190edd194c338535a79618928568"
format_stamp: "Formatted at 2026-09-25 05:00:59 on dist-test-slave-5n86"
server_key: "9d88f1a5886654abcdbac1a94e101445"
server_key_iv: "ed58c98d7d61b3fa9b39247f864e6f03"
server_key_version: "encryptionkey@0"
I20260925 05:01:00.007546 2671 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260925 05:01:00.010936 2686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:00.012322 2671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:01:00.012490 2671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "9060190edd194c338535a79618928568"
format_stamp: "Formatted at 2026-09-25 05:00:59 on dist-test-slave-5n86"
server_key: "9d88f1a5886654abcdbac1a94e101445"
server_key_iv: "ed58c98d7d61b3fa9b39247f864e6f03"
server_key_version: "encryptionkey@0"
I20260925 05:01:00.012742 2671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:00.034552 2671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:00.038866 2671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:00.039148 2671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:00.040083 2671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:00.041394 2671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:00.041496 2671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:00.041628 2671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:00.041702 2671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:00.074187 2671 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:35353
I20260925 05:01:00.074240 2798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:35353 every 8 connection(s)
I20260925 05:01:00.075441 2671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:01:00.083365 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 2671
I20260925 05:01:00.083540 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance
I20260925 05:01:00.084048 31708 external_mini_cluster.cc:1428] Setting key b7a2db8fa24c7e81e790eb83643a3e6f
Sep 25 05:01:00 dist-test-slave-5n86 krb5kdc[2297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312459, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:01:00.094981 2534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:00.078031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:41747 (local address 127.30.247.62:37731)
0925 05:01:00.078203 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:00.078207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:00.079356 (+ 1149us) server_negotiation.cc:407] Connection header received
0925 05:01:00.081215 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:00.081221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:00.081313 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:00.081424 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:00.082851 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:00.083703 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:00.085572 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:00.086014 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:00.086145 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:00.089547 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:00.089552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:00.089555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:00.089585 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:00.092023 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:00.092861 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:00.092865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:00.092867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:00.092978 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:00.093391 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:00.093394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:00.093396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:00.093643 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:00.093754 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:00.094514 (+ 760us) server_negotiation.cc:299] Negotiation successful
0925 05:01:00.094710 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 05:01:00.095775 2801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:00.078504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37731 (local address 127.30.247.3:41747)
0925 05:01:00.079200 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:00.079235 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:00.080960 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:00.081658 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:00.081666 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:00.082139 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:00.082686 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:00.082703 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:00.083853 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:00.083863 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:00.085439 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:00.085450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:00.086231 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:00.086237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:00.086437 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:00.087184 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:00.087207 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:00.089356 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:00.092162 (+ 2806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:00.092169 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:00.092181 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:00.092749 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:00.093069 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:00.093073 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:00.093075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:00.093263 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:00.093791 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:00.093800 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:00.094143 (+ 343us) client_negotiation.cc:769] Sending connection context
0925 05:01:00.094380 (+ 237us) client_negotiation.cc:240] Negotiation successful
0925 05:01:00.094750 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":274,"threads_started":1}
I20260925 05:01:00.097273 2799 heartbeater.cc:347] Connected to a master server at 127.30.247.62:37731
I20260925 05:01:00.097585 2799 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:00.098428 2799 heartbeater.cc:511] Master 127.30.247.62:37731 requested a full tablet report, sending...
I20260925 05:01:00.099938 2342 ts_manager.cc:194] Registered new tserver with Master: 9060190edd194c338535a79618928568 (127.30.247.3:35353)
I20260925 05:01:00.100807 2342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.3@KRBTEST.COM'} at 127.30.247.3:41747
I20260925 05:01:00.111402 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 05:01:00 dist-test-slave-5n86 krb5kdc[2297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312460, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 05:01:00 dist-test-slave-5n86 krb5kdc[2297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312460, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:01:00.159166 2534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:00.145767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51138 (local address 127.30.247.62:37731)
0925 05:01:00.146022 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:00.146029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:00.146268 (+ 239us) server_negotiation.cc:407] Connection header received
0925 05:01:00.146408 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:00.146412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:00.146484 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:00.146571 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:00.147862 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:00.148741 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:00.149995 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:00.150402 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:00.150511 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:00.154310 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:00.154315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:00.154318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:00.154349 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:00.156768 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:00.157325 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:00.157329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:00.157331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:00.157467 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:00.157818 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:00.157822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:00.157824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:00.158212 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:00.158390 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:00.158661 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 05:01:00.158824 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260925 05:01:00.159751 2342 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51138
I20260925 05:01:00.161412 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 2403
I20260925 05:01:00.165863 2532 heartbeater.cc:503] Master 127.30.247.62:37731 was elected leader, sending a full tablet report...
Sep 25 05:01:00 dist-test-slave-5n86 krb5kdc[2297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312460, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.30.247.1@KRBTEST.COM
I20260925 05:01:00.176841 2810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:00.162285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37496 (local address 127.30.247.1:42567)
0925 05:01:00.162844 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:00.162851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:00.162875 (+ 24us) server_negotiation.cc:407] Connection header received
0925 05:01:00.162926 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:00.162932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:00.163129 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:00.163250 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:00.164243 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:00.165351 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:00.166726 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:00.167551 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:00.167659 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:00.171888 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:00.171898 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:00.171912 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:00.171946 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:00.174555 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:00.175107 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:00.175113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:00.175115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:00.175228 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:00.175554 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:00.175559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:00.175561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:00.175883 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:00.176106 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:00.176282 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 05:01:00.176461 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":272,"threads_started":1}
W20260925 05:01:00.178016 2527 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37496
W20260925 05:01:00.179101 31708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 2403: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:01:00.179318 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 2403
I20260925 05:01:00.194792 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 2538
Sep 25 05:01:00 dist-test-slave-5n86 krb5kdc[2297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312460, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.30.247.2@KRBTEST.COM
I20260925 05:01:00.209777 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:00.195583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55484 (local address 127.30.247.2:34851)
0925 05:01:00.196095 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:00.196102 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:00.196126 (+ 24us) server_negotiation.cc:407] Connection header received
0925 05:01:00.196180 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:00.196187 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:00.196390 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:00.196514 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:00.197663 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:00.198749 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:00.199914 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:00.200411 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:00.200501 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:00.204494 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:00.204503 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:00.204517 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:00.204888 (+ 371us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:00.207409 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:00.207970 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:00.207976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:00.207978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:00.208095 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:00.208393 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:00.208395 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:00.208397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:00.208871 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:00.209087 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:00.209323 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 05:01:00.209504 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":285,"threads_started":1}
W20260925 05:01:00.210948 2661 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55484
W20260925 05:01:00.211963 31708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 2538: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:01:00.212101 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 2538
I20260925 05:01:00.226750 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 2671
Sep 25 05:01:00 dist-test-slave-5n86 krb5kdc[2297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312460, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.30.247.3@KRBTEST.COM
I20260925 05:01:00.242151 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:00.227618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37948 (local address 127.30.247.3:35353)
0925 05:01:00.228284 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:00.228292 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:00.228320 (+ 28us) server_negotiation.cc:407] Connection header received
0925 05:01:00.228374 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:00.228380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:00.228633 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:00.228764 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:00.229908 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:00.230907 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:00.232028 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:00.232666 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:00.232760 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:00.236634 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:00.236644 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:00.236658 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:00.236692 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:00.239597 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:00.240165 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:00.240171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:00.240173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:00.240285 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:00.240733 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:00.240736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:00.240738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:00.241053 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:00.241255 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:00.241589 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 05:01:00.241774 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":385,"threads_started":1}
W20260925 05:01:00.243389 2794 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37948
W20260925 05:01:00.244398 31708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 2671: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:01:00.244647 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 2671
I20260925 05:01:00.259752 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 2310
I20260925 05:01:00.260874 2372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:00.362632 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 2310
2026-09-25T05:01:00Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3460 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:01:00 dist-test-slave-5n86 krb5kdc[2816](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:01:00 dist-test-slave-5n86 krb5kdc[2816](info): set up 2 sockets
Sep 25 05:01:00 dist-test-slave-5n86 krb5kdc[2816](info): commencing operation
krb5kdc: starting...
W20260925 05:01:01.445058 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:01:01 dist-test-slave-5n86 krb5kdc[2816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:01:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:01:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:01:01.650136 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:46377
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:42861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:46377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:01.962733 2829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:01.963119 2829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:01.963277 2829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:01.970467 2829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:01:01.970557 2829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:01.970651 2829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:01.970742 2829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:01:01.970821 2829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:01:01.980881 2829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:42861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:46377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:46377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:01.982090 2829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:01.983312 2829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:01.994743 2837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:01.994954 2835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:01.995258 2834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:01.995918 2829 server_base.cc:1034] running on GCE node
I20260925 05:01:01.996719 2829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:01.997987 2829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:01.999214 2829 hybrid_clock.cc:648] HybridClock initialized: now 1790312461999183 us; error 45 us; skew 500 ppm
I20260925 05:01:01.999543 2829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:02 dist-test-slave-5n86 krb5kdc[2816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312462, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:02.007705 2829 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:01:02.009152 2829 webserver.cc:460] Webserver started at http://127.30.247.62:38713/ using document root <none> and password file <none>
I20260925 05:01:02.009817 2829 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:02.009958 2829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:02.010260 2829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:02.013203 2829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "71d3385a1eff47909fd5d2754fb05cc0"
format_stamp: "Formatted at 2026-09-25 05:01:02 on dist-test-slave-5n86"
server_key: "d012c1fac3e1ccf0838ef96c6dc8d609"
server_key_iv: "9c77127a5b8aad124565878358a0d2a8"
server_key_version: "encryptionkey@0"
I20260925 05:01:02.013993 2829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "71d3385a1eff47909fd5d2754fb05cc0"
format_stamp: "Formatted at 2026-09-25 05:01:02 on dist-test-slave-5n86"
server_key: "d012c1fac3e1ccf0838ef96c6dc8d609"
server_key_iv: "9c77127a5b8aad124565878358a0d2a8"
server_key_version: "encryptionkey@0"
I20260925 05:01:02.018863 2829 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 05:01:02.022094 2844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:02.023325 2829 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 05:01:02.023501 2829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "71d3385a1eff47909fd5d2754fb05cc0"
format_stamp: "Formatted at 2026-09-25 05:01:02 on dist-test-slave-5n86"
server_key: "d012c1fac3e1ccf0838ef96c6dc8d609"
server_key_iv: "9c77127a5b8aad124565878358a0d2a8"
server_key_version: "encryptionkey@0"
I20260925 05:01:02.023682 2829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:02.035895 2829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:02.039608 2829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:02.039881 2829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:02.057548 2829 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:46377
I20260925 05:01:02.057571 2895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:46377 every 8 connection(s)
I20260925 05:01:02.058822 2829 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:01:02.062377 2896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:02.066752 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 2829
I20260925 05:01:02.066980 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:01:02.067749 31708 external_mini_cluster.cc:1428] Setting key fa38ebd0e9cbe6daa9a4d34647e2fc23
I20260925 05:01:02.073542 2896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71d3385a1eff47909fd5d2754fb05cc0: Bootstrap starting.
I20260925 05:01:02.076408 2896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71d3385a1eff47909fd5d2754fb05cc0: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:02.077404 2896 log.cc:824] T 00000000000000000000000000000000 P 71d3385a1eff47909fd5d2754fb05cc0: Log is configured to *not* fsync() on all Append() calls
I20260925 05:01:02.079986 2896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71d3385a1eff47909fd5d2754fb05cc0: No bootstrap required, opened a new log
Sep 25 05:01:02 dist-test-slave-5n86 krb5kdc[2816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:01:02.089146 2896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71d3385a1eff47909fd5d2754fb05cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d3385a1eff47909fd5d2754fb05cc0" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46377 } }
I20260925 05:01:02.089533 2896 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71d3385a1eff47909fd5d2754fb05cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:02.089679 2896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71d3385a1eff47909fd5d2754fb05cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d3385a1eff47909fd5d2754fb05cc0, State: Initialized, Role: FOLLOWER
I20260925 05:01:02.089864 2899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:02.069776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33528 (local address 127.30.247.62:46377)
0925 05:01:02.070527 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:02.070539 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:02.070576 (+ 37us) server_negotiation.cc:407] Connection header received
0925 05:01:02.072248 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:02.072267 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:02.072693 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:02.073085 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:02.074513 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:02.075785 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:02.077128 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:02.077863 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:02.078073 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:02.082235 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:02.082245 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:02.082258 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:02.082291 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:02.085663 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:02.086478 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:02.086485 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:02.086494 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:02.086646 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:02.087035 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:02.087038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:02.087041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:02.087560 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:02.087838 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:02.088246 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 05:01:02.088669 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":337,"threads_started":1}
I20260925 05:01:02.090404 2896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71d3385a1eff47909fd5d2754fb05cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d3385a1eff47909fd5d2754fb05cc0" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46377 } }
I20260925 05:01:02.090659 2896 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71d3385a1eff47909fd5d2754fb05cc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:01:02.090809 2896 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71d3385a1eff47909fd5d2754fb05cc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:01:02.090992 2896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71d3385a1eff47909fd5d2754fb05cc0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:02.093235 2896 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71d3385a1eff47909fd5d2754fb05cc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d3385a1eff47909fd5d2754fb05cc0" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46377 } }
I20260925 05:01:02.093715 2896 leader_election.cc:302] T 00000000000000000000000000000000 P 71d3385a1eff47909fd5d2754fb05cc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71d3385a1eff47909fd5d2754fb05cc0; no voters:
I20260925 05:01:02.094300 2896 leader_election.cc:288] T 00000000000000000000000000000000 P 71d3385a1eff47909fd5d2754fb05cc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:01:02.094450 2901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71d3385a1eff47909fd5d2754fb05cc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:01:02.095158 2901 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71d3385a1eff47909fd5d2754fb05cc0 [term 1 LEADER]: Becoming Leader. State: Replica: 71d3385a1eff47909fd5d2754fb05cc0, State: Running, Role: LEADER
I20260925 05:01:02.095683 2901 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71d3385a1eff47909fd5d2754fb05cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d3385a1eff47909fd5d2754fb05cc0" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46377 } }
I20260925 05:01:02.096342 2896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71d3385a1eff47909fd5d2754fb05cc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:01:02.098647 2902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71d3385a1eff47909fd5d2754fb05cc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71d3385a1eff47909fd5d2754fb05cc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d3385a1eff47909fd5d2754fb05cc0" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46377 } } }
I20260925 05:01:02.099004 2902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71d3385a1eff47909fd5d2754fb05cc0 [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:02.098711 2903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71d3385a1eff47909fd5d2754fb05cc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71d3385a1eff47909fd5d2754fb05cc0. Latest consensus state: current_term: 1 leader_uuid: "71d3385a1eff47909fd5d2754fb05cc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d3385a1eff47909fd5d2754fb05cc0" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46377 } } }
I20260925 05:01:02.099107 2903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71d3385a1eff47909fd5d2754fb05cc0 [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:02.104676 2908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:01:02.108457 2908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:01:02.116719 2908 catalog_manager.cc:1347] Generated new cluster ID: 308510d3dc2f414bb5980d70cd03cb97
I20260925 05:01:02.116811 2908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:01:02.134692 2908 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:01:02.135878 2908 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:01:02.143077 2908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71d3385a1eff47909fd5d2754fb05cc0: Generated new TSK 0
I20260925 05:01:02.143821 2908 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
I20260925 05:01:02.196269 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:0
--local_ip_for_outbound_sockets=127.30.247.1
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:46377
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:42861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:02.506412 2922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:02.506692 2922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:02.506860 2922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:02.513677 2922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:02.513821 2922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:02.513984 2922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:01:02.522938 2922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:42861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:46377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:02.524039 2922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:02.525259 2922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:02.537833 2927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:02.537794 2928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:02.537868 2930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:02.538870 2922 server_base.cc:1034] running on GCE node
I20260925 05:01:02.539422 2922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:02.540283 2922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:02.541496 2922 hybrid_clock.cc:648] HybridClock initialized: now 1790312462541406 us; error 108 us; skew 500 ppm
I20260925 05:01:02.541826 2922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:02 dist-test-slave-5n86 krb5kdc[2816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312462, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:02.544965 2922 init.cc:377] Logged in from keytab as kudu/127.30.247.1@KRBTEST.COM (short username kudu)
I20260925 05:01:02.546376 2922 webserver.cc:460] Webserver started at http://127.30.247.1:38539/ using document root <none> and password file <none>
I20260925 05:01:02.547025 2922 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:02.547166 2922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:02.547458 2922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:02.550300 2922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-0/data/instance:
uuid: "eb1b549919fd4be292f7cc3d89495062"
format_stamp: "Formatted at 2026-09-25 05:01:02 on dist-test-slave-5n86"
server_key: "826b1d2f4c8d2f623859ece50307a283"
server_key_iv: "4b1088513ca7ec1356f2e7e15b525a9c"
server_key_version: "encryptionkey@0"
I20260925 05:01:02.551039 2922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance:
uuid: "eb1b549919fd4be292f7cc3d89495062"
format_stamp: "Formatted at 2026-09-25 05:01:02 on dist-test-slave-5n86"
server_key: "826b1d2f4c8d2f623859ece50307a283"
server_key_iv: "4b1088513ca7ec1356f2e7e15b525a9c"
server_key_version: "encryptionkey@0"
I20260925 05:01:02.555752 2922 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 05:01:02.558995 2937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:02.560177 2922 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 05:01:02.560348 2922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "eb1b549919fd4be292f7cc3d89495062"
format_stamp: "Formatted at 2026-09-25 05:01:02 on dist-test-slave-5n86"
server_key: "826b1d2f4c8d2f623859ece50307a283"
server_key_iv: "4b1088513ca7ec1356f2e7e15b525a9c"
server_key_version: "encryptionkey@0"
I20260925 05:01:02.560542 2922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:02.608884 2922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:02.612828 2922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:02.613097 2922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:02.614015 2922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:02.615314 2922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:02.615417 2922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:02.615558 2922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:02.615628 2922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:02.648703 2922 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:35615
I20260925 05:01:02.648763 3049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:35615 every 8 connection(s)
I20260925 05:01:02.649991 2922 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:01:02.654330 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 2922
I20260925 05:01:02.654558 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance
I20260925 05:01:02.655416 31708 external_mini_cluster.cc:1428] Setting key a841370566a705481273c6cf292d88a9
Sep 25 05:01:02 dist-test-slave-5n86 krb5kdc[2816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312462, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:01:02.673396 3052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:02.652510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:36301 (local address 127.30.247.62:46377)
0925 05:01:02.652973 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:02.652978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:02.654071 (+ 1093us) server_negotiation.cc:407] Connection header received
0925 05:01:02.656699 (+ 2628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:02.656704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:02.656798 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:02.656917 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:02.658937 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:02.659684 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:02.662147 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:02.662555 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:02.662655 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:02.666331 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:02.666337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:02.666344 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:02.666388 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:02.669980 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:02.670693 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:02.670701 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:02.670707 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:02.670861 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:02.671261 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:02.671269 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:02.671275 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:02.671666 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:02.671830 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:02.672599 (+ 769us) server_negotiation.cc:299] Negotiation successful
0925 05:01:02.672878 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":230,"threads_started":1}
I20260925 05:01:02.673978 3053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:02.653142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:46377 (local address 127.30.247.1:36301)
0925 05:01:02.653822 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:02.653931 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:02.656319 (+ 2388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:02.657178 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:02.657190 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:02.657911 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:02.658740 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:02.658769 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:02.659841 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:02.659849 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:02.661997 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:02.662014 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:02.662784 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:02.662793 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:02.663109 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:02.663874 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:02.663900 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:02.666112 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:02.670130 (+ 4018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:02.670137 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:02.670150 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:02.670553 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:02.670964 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:02.670967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:02.670969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:02.671138 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:02.671822 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:02.671828 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:02.672153 (+ 325us) client_negotiation.cc:769] Sending connection context
0925 05:01:02.672397 (+ 244us) client_negotiation.cc:240] Negotiation successful
0925 05:01:02.672860 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":264,"threads_started":1}
WARNING: no policy specified for kudu/127.30.247.2@KRBTEST.COM; defaulting to no policy
I20260925 05:01:02.675606 3050 heartbeater.cc:347] Connected to a master server at 127.30.247.62:46377
I20260925 05:01:02.675946 3050 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:02.676833 3050 heartbeater.cc:511] Master 127.30.247.62:46377 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.2@KRBTEST.COM" created.
I20260925 05:01:02.679155 2861 ts_manager.cc:194] Registered new tserver with Master: eb1b549919fd4be292f7cc3d89495062 (127.30.247.1:35615)
I20260925 05:01:02.680673 2861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.1@KRBTEST.COM'} at 127.30.247.1:36301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
I20260925 05:01:02.704180 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:0
--local_ip_for_outbound_sockets=127.30.247.2
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:46377
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:42861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:03.011602 3056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:03.011845 3056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:03.011932 3056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:03.018663 3056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:03.018754 3056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:03.018858 3056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:01:03.027864 3056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:42861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:46377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:03.029040 3056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:03.030297 3056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:03.042431 3064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:03.042538 3062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:03.042631 3061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:03.043164 3056 server_base.cc:1034] running on GCE node
I20260925 05:01:03.043812 3056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:03.044730 3056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:03.045944 3056 hybrid_clock.cc:648] HybridClock initialized: now 1790312463045895 us; error 54 us; skew 500 ppm
I20260925 05:01:03.046231 3056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:03 dist-test-slave-5n86 krb5kdc[2816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312463, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:03.049367 3056 init.cc:377] Logged in from keytab as kudu/127.30.247.2@KRBTEST.COM (short username kudu)
I20260925 05:01:03.050675 3056 webserver.cc:460] Webserver started at http://127.30.247.2:43131/ using document root <none> and password file <none>
I20260925 05:01:03.051321 3056 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:03.051445 3056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:03.051748 3056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:03.054487 3056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-1/data/instance:
uuid: "6a415538c2af47e8a80c4ec55561f98a"
format_stamp: "Formatted at 2026-09-25 05:01:03 on dist-test-slave-5n86"
server_key: "9af96b7f0c68192b1ed70a4bec477350"
server_key_iv: "6f4ccb833539532265000fde9f1adbda"
server_key_version: "encryptionkey@0"
I20260925 05:01:03.055217 3056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance:
uuid: "6a415538c2af47e8a80c4ec55561f98a"
format_stamp: "Formatted at 2026-09-25 05:01:03 on dist-test-slave-5n86"
server_key: "9af96b7f0c68192b1ed70a4bec477350"
server_key_iv: "6f4ccb833539532265000fde9f1adbda"
server_key_version: "encryptionkey@0"
I20260925 05:01:03.060261 3056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 05:01:03.063544 3071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:03.064961 3056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:01:03.065119 3056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "6a415538c2af47e8a80c4ec55561f98a"
format_stamp: "Formatted at 2026-09-25 05:01:03 on dist-test-slave-5n86"
server_key: "9af96b7f0c68192b1ed70a4bec477350"
server_key_iv: "6f4ccb833539532265000fde9f1adbda"
server_key_version: "encryptionkey@0"
I20260925 05:01:03.065299 3056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:03.079140 3056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:03.082859 3056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:03.083145 3056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:03.083992 3056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:03.085296 3056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:03.085368 3056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:03.085534 3056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:03.085588 3056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:03.116245 3056 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:39679
I20260925 05:01:03.116276 3183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:39679 every 8 connection(s)
I20260925 05:01:03.117980 3056 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:01:03.121145 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 3056
I20260925 05:01:03.121340 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance
I20260925 05:01:03.121829 31708 external_mini_cluster.cc:1428] Setting key b0d341552642330134fd2061c66d597a
Sep 25 05:01:03 dist-test-slave-5n86 krb5kdc[2816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312463, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:01:03.139325 3052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:03.121160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:44403 (local address 127.30.247.62:46377)
0925 05:01:03.121357 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:03.121361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:03.122850 (+ 1489us) server_negotiation.cc:407] Connection header received
0925 05:01:03.124872 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:03.124878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:03.124959 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:03.125072 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:03.126564 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:03.127366 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:03.129059 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:03.129463 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:03.129563 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:03.133608 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:03.133613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:03.133617 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:03.133646 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:03.136299 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:03.137116 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:03.137122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:03.137127 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:03.137255 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:03.137684 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:03.137693 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:03.137695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:03.137941 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:03.138096 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:03.138750 (+ 654us) server_negotiation.cc:299] Negotiation successful
0925 05:01:03.138969 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
WARNING: no policy specified for kudu/127.30.247.3@KRBTEST.COM; defaulting to no policy
I20260925 05:01:03.140300 3186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:03.121818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:46377 (local address 127.30.247.2:44403)
0925 05:01:03.122674 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:03.122722 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:03.124619 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:03.125306 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:03.125314 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:03.125803 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:03.126412 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:03.126432 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:03.127527 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:03.127539 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:03.128894 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:03.128928 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:03.129677 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:03.129684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:03.130009 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:03.130725 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:03.130759 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:03.133396 (+ 2637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:03.136431 (+ 3035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:03.136440 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:03.136455 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:03.136989 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:03.137369 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:03.137375 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:03.137378 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:03.137564 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:03.138069 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:03.138076 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:03.138408 (+ 332us) client_negotiation.cc:769] Sending connection context
0925 05:01:03.138653 (+ 245us) client_negotiation.cc:240] Negotiation successful
0925 05:01:03.139169 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":335,"threads_started":1}
I20260925 05:01:03.141693 3184 heartbeater.cc:347] Connected to a master server at 127.30.247.62:46377
I20260925 05:01:03.141993 3184 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.3@KRBTEST.COM" created.
I20260925 05:01:03.142778 3184 heartbeater.cc:511] Master 127.30.247.62:46377 requested a full tablet report, sending...
I20260925 05:01:03.144265 2861 ts_manager.cc:194] Registered new tserver with Master: 6a415538c2af47e8a80c4ec55561f98a (127.30.247.2:39679)
I20260925 05:01:03.145503 2861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.2@KRBTEST.COM'} at 127.30.247.2:44403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
I20260925 05:01:03.164314 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:0
--local_ip_for_outbound_sockets=127.30.247.3
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:46377
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:42861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:03.469544 3189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:03.469789 3189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:03.469884 3189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:03.476624 3189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:03.476712 3189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:03.476820 3189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:01:03.485692 3189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:42861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:46377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:03.486824 3189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:03.488013 3189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:03.500450 3195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:03.500485 3197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:03.500618 3194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:03.501192 3189 server_base.cc:1034] running on GCE node
I20260925 05:01:03.501750 3189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:03.502686 3189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:03.503878 3189 hybrid_clock.cc:648] HybridClock initialized: now 1790312463503822 us; error 51 us; skew 500 ppm
I20260925 05:01:03.504171 3189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:03 dist-test-slave-5n86 krb5kdc[2816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312463, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:03.507236 3189 init.cc:377] Logged in from keytab as kudu/127.30.247.3@KRBTEST.COM (short username kudu)
I20260925 05:01:03.508668 3189 webserver.cc:460] Webserver started at http://127.30.247.3:34323/ using document root <none> and password file <none>
I20260925 05:01:03.509259 3189 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:03.509357 3189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:03.509600 3189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:03.512213 3189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-2/data/instance:
uuid: "f26d61fbdb6f42d4b901a140c8dcf175"
format_stamp: "Formatted at 2026-09-25 05:01:03 on dist-test-slave-5n86"
server_key: "a921eb6e4cc222ba1cc6b37d75609640"
server_key_iv: "de1c9b68e9a4c87744a88d4652db1df4"
server_key_version: "encryptionkey@0"
I20260925 05:01:03.512964 3189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance:
uuid: "f26d61fbdb6f42d4b901a140c8dcf175"
format_stamp: "Formatted at 2026-09-25 05:01:03 on dist-test-slave-5n86"
server_key: "a921eb6e4cc222ba1cc6b37d75609640"
server_key_iv: "de1c9b68e9a4c87744a88d4652db1df4"
server_key_version: "encryptionkey@0"
I20260925 05:01:03.517570 3189 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 05:01:03.520851 3204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:03.522065 3189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 05:01:03.522235 3189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "f26d61fbdb6f42d4b901a140c8dcf175"
format_stamp: "Formatted at 2026-09-25 05:01:03 on dist-test-slave-5n86"
server_key: "a921eb6e4cc222ba1cc6b37d75609640"
server_key_iv: "de1c9b68e9a4c87744a88d4652db1df4"
server_key_version: "encryptionkey@0"
I20260925 05:01:03.522441 3189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:03.554412 3189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:03.558698 3189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:03.558983 3189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:03.559919 3189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:03.561235 3189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:03.561336 3189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:03.561484 3189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:03.561553 3189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:03.593343 3189 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:46867
I20260925 05:01:03.593425 3316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:46867 every 8 connection(s)
I20260925 05:01:03.594586 3189 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:01:03.599517 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 3189
I20260925 05:01:03.599706 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance
I20260925 05:01:03.600107 31708 external_mini_cluster.cc:1428] Setting key 830bc14466e8089036ec99575f4abc6a
Sep 25 05:01:03 dist-test-slave-5n86 krb5kdc[2816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312463, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:01:03.613157 3052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:03.597061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:53249 (local address 127.30.247.62:46377)
0925 05:01:03.597231 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:03.597235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:03.598469 (+ 1234us) server_negotiation.cc:407] Connection header received
0925 05:01:03.600355 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:03.600360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:03.600472 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:03.600618 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:03.602329 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:03.603300 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:03.604829 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:03.605224 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:03.605348 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:03.608409 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:03.608414 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:03.608417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:03.608445 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:03.610715 (+ 2270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:03.611381 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:03.611384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:03.611387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:03.611486 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:03.611832 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:03.611835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:03.611837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:03.612066 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:03.612168 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:03.612775 (+ 607us) server_negotiation.cc:299] Negotiation successful
0925 05:01:03.612924 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 05:01:03.614342 3319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:03.597613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:46377 (local address 127.30.247.3:53249)
0925 05:01:03.598318 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:03.598355 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:03.600079 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:03.600835 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:03.600843 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:03.601465 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:03.602179 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:03.602196 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:03.603447 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:03.603458 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:03.604689 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:03.604700 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:03.605451 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:03.605458 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:03.605667 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:03.606219 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:03.606238 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:03.608235 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:03.610843 (+ 2608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:03.610849 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:03.610859 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:03.611287 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:03.611572 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:03.611575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:03.611577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:03.611735 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:03.612165 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:03.612170 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:03.612433 (+ 263us) client_negotiation.cc:769] Sending connection context
0925 05:01:03.612697 (+ 264us) client_negotiation.cc:240] Negotiation successful
0925 05:01:03.613113 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":288,"threads_started":1}
I20260925 05:01:03.615595 3317 heartbeater.cc:347] Connected to a master server at 127.30.247.62:46377
I20260925 05:01:03.615865 3317 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:03.616514 3317 heartbeater.cc:511] Master 127.30.247.62:46377 requested a full tablet report, sending...
I20260925 05:01:03.617802 2861 ts_manager.cc:194] Registered new tserver with Master: f26d61fbdb6f42d4b901a140c8dcf175 (127.30.247.3:46867)
I20260925 05:01:03.618556 2861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.3@KRBTEST.COM'} at 127.30.247.3:53249
I20260925 05:01:03.626989 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 05:01:03 dist-test-slave-5n86 krb5kdc[2816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 05:01:03 dist-test-slave-5n86 krb5kdc[2816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.1@KRBTEST.COM
I20260925 05:01:03.673421 3324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:03.658866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33646 (local address 127.30.247.1:35615)
0925 05:01:03.659467 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:03.659477 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:03.659509 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:01:03.659590 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:03.659596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:03.659810 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:03.659980 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:03.661371 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:03.662358 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:03.663605 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:03.664163 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:03.664288 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:03.667964 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:03.667975 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:03.667986 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:03.668015 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:03.670751 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:03.671393 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:03.671406 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:03.671408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:03.671529 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:03.671898 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:03.671904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:03.671907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:03.672216 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:03.672524 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:03.672840 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 05:01:03.673086 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":299,"threads_started":1}
I20260925 05:01:03.683461 3050 heartbeater.cc:503] Master 127.30.247.62:46377 was elected leader, sending a full tablet report...
Sep 25 05:01:03 dist-test-slave-5n86 krb5kdc[2816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:01:03.689437 3052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:03.677382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33532 (local address 127.30.247.62:46377)
0925 05:01:03.677583 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:03.677589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:03.677724 (+ 135us) server_negotiation.cc:407] Connection header received
0925 05:01:03.677847 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:03.677851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:03.677912 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:03.677990 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:03.679093 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:03.679832 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:03.681036 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:03.681414 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:03.681506 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:03.684989 (+ 3483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:03.684996 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:03.685007 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:03.685048 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:03.687329 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:03.687883 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:03.687887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:03.687889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:03.688006 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:03.688284 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:03.688287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:03.688289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:03.688633 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:03.688833 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:03.689082 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 05:01:03.689228 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 05:01:03.689941 2861 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33532
I20260925 05:01:03.690869 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 2922
I20260925 05:01:03.703018 3324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:03.691651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33660 (local address 127.30.247.1:35615)
0925 05:01:03.691847 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:03.691855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:03.692115 (+ 260us) server_negotiation.cc:407] Connection header received
0925 05:01:03.692296 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:03.692301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:03.692375 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:03.692476 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:03.693597 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:03.694262 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:03.695456 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:03.695861 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:03.695993 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:03.698170 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:03.698176 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:03.698182 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:03.698221 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:03.700499 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:03.701135 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:03.701141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:03.701145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:03.701273 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:03.701656 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:03.701661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:03.701665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:03.701947 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:03.702185 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:03.702480 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 05:01:03.702714 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 05:01:03.703538 3045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:03.784493 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 2922
I20260925 05:01:03.800344 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 3056
Sep 25 05:01:03 dist-test-slave-5n86 krb5kdc[2816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.2@KRBTEST.COM
I20260925 05:01:03.815901 3329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:03.801227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45798 (local address 127.30.247.2:39679)
0925 05:01:03.801833 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:03.801840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:03.801866 (+ 26us) server_negotiation.cc:407] Connection header received
0925 05:01:03.801926 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:03.801932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:03.802146 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:03.802334 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:03.803511 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:03.804517 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:03.805838 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:03.806393 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:03.806523 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:03.810448 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:03.810459 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:03.810472 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:03.810502 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:03.813245 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:03.813880 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:03.813887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:03.813890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:03.814008 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:03.814375 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:03.814381 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:03.814385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:03.814703 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:03.815014 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:03.815313 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 05:01:03.815564 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":322,"threads_started":1}
I20260925 05:01:03.817165 3179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:03.898214 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 3056
I20260925 05:01:03.913777 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 3189
Sep 25 05:01:03 dist-test-slave-5n86 krb5kdc[2816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.3@KRBTEST.COM
I20260925 05:01:03.930083 3331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:03.914793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51588 (local address 127.30.247.3:46867)
0925 05:01:03.915470 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:03.915477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:03.915507 (+ 30us) server_negotiation.cc:407] Connection header received
0925 05:01:03.915568 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:03.915574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:03.915797 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:03.915931 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:03.917228 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:03.918206 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:03.919428 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:03.919971 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:03.920067 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:03.924241 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:03.924252 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:03.924265 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:03.924295 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:03.927341 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:03.928014 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:03.928022 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:03.928024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:03.928142 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:03.928515 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:03.928521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:03.928524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:03.928918 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:03.929234 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:03.929481 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 05:01:03.929742 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":379,"threads_started":1}
I20260925 05:01:03.931342 3312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:04.021992 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 3189
I20260925 05:01:04.037312 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 2829
I20260925 05:01:04.038584 2891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:04.141265 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 2829
2026-09-25T05:01:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3778 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:01:04 dist-test-slave-5n86 krb5kdc[3336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:01:04 dist-test-slave-5n86 krb5kdc[3336](info): set up 2 sockets
Sep 25 05:01:04 dist-test-slave-5n86 krb5kdc[3336](info): commencing operation
krb5kdc: starting...
W20260925 05:01:05.229194 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:01:05 dist-test-slave-5n86 krb5kdc[3336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:01:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:01:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:01:05.412925 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:39009
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:44107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:39009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:05.714716 3349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:05.714972 3349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:05.715132 3349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:05.721930 3349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:01:05.722018 3349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:05.722106 3349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:05.722210 3349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:01:05.722296 3349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:01:05.731627 3349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:44107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:39009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:39009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:05.732812 3349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:05.734076 3349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:05.745282 3355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:05.745209 3357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:05.745342 3354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:05.746090 3349 server_base.cc:1034] running on GCE node
I20260925 05:01:05.746832 3349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:05.748188 3349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:05.749441 3349 hybrid_clock.cc:648] HybridClock initialized: now 1790312465749395 us; error 42 us; skew 500 ppm
I20260925 05:01:05.749820 3349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:05 dist-test-slave-5n86 krb5kdc[3336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312465, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:05.752947 3349 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:01:05.754145 3349 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:05.754276 3349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:05.754662 3349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:05.757566 3349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "a5c98d009c314e9087546177a733eb4b"
format_stamp: "Formatted at 2026-09-25 05:01:05 on dist-test-slave-5n86"
server_key: "76592cd81adfdc0d8ba78fca4dc64582"
server_key_iv: "33e6db9b9064d9616f05d525cf636d5f"
server_key_version: "encryptionkey@0"
I20260925 05:01:05.758343 3349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "a5c98d009c314e9087546177a733eb4b"
format_stamp: "Formatted at 2026-09-25 05:01:05 on dist-test-slave-5n86"
server_key: "76592cd81adfdc0d8ba78fca4dc64582"
server_key_iv: "33e6db9b9064d9616f05d525cf636d5f"
server_key_version: "encryptionkey@0"
I20260925 05:01:05.763151 3349 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 05:01:05.766461 3363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:05.767741 3349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:01:05.767920 3349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "a5c98d009c314e9087546177a733eb4b"
format_stamp: "Formatted at 2026-09-25 05:01:05 on dist-test-slave-5n86"
server_key: "76592cd81adfdc0d8ba78fca4dc64582"
server_key_iv: "33e6db9b9064d9616f05d525cf636d5f"
server_key_version: "encryptionkey@0"
I20260925 05:01:05.768134 3349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:05.782332 3349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:05.785933 3349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:05.786183 3349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:05.803768 3349 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:39009
I20260925 05:01:05.803813 3414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:39009 every 8 connection(s)
I20260925 05:01:05.804644 3349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:01:05.807605 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 3349
I20260925 05:01:05.807857 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:01:05.808211 3415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:05.808508 31708 external_mini_cluster.cc:1428] Setting key 5c7306f230f5f627a18da5e067ec6fa8
I20260925 05:01:05.822077 3415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5c98d009c314e9087546177a733eb4b: Bootstrap starting.
Sep 25 05:01:05 dist-test-slave-5n86 krb5kdc[3336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:01:05.825224 3415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5c98d009c314e9087546177a733eb4b: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:05.826244 3415 log.cc:824] T 00000000000000000000000000000000 P a5c98d009c314e9087546177a733eb4b: Log is configured to *not* fsync() on all Append() calls
I20260925 05:01:05.828927 3415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5c98d009c314e9087546177a733eb4b: No bootstrap required, opened a new log
I20260925 05:01:05.831122 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:05.810428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48110 (local address 127.30.247.62:39009)
0925 05:01:05.811272 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:05.811283 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:05.811324 (+ 41us) server_negotiation.cc:407] Connection header received
0925 05:01:05.812939 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:05.812960 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:05.813328 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:05.813640 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:05.815486 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:05.816832 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:05.818664 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:05.819361 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:05.819499 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:05.824010 (+ 4511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:05.824017 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:05.824034 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:05.824073 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:05.827172 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:05.827912 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:05.827920 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:05.827934 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:05.828090 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:05.828453 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:05.828458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:05.828462 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:05.829055 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:05.829351 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:05.829782 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 05:01:05.830179 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":403,"threads_started":1}
I20260925 05:01:05.836614 3415 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5c98d009c314e9087546177a733eb4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5c98d009c314e9087546177a733eb4b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 39009 } }
I20260925 05:01:05.836990 3415 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5c98d009c314e9087546177a733eb4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:05.837101 3415 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5c98d009c314e9087546177a733eb4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5c98d009c314e9087546177a733eb4b, State: Initialized, Role: FOLLOWER
I20260925 05:01:05.837579 3415 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5c98d009c314e9087546177a733eb4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5c98d009c314e9087546177a733eb4b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 39009 } }
I20260925 05:01:05.837766 3415 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5c98d009c314e9087546177a733eb4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:01:05.837862 3415 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5c98d009c314e9087546177a733eb4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:01:05.837984 3415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5c98d009c314e9087546177a733eb4b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:05.840238 3415 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5c98d009c314e9087546177a733eb4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5c98d009c314e9087546177a733eb4b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 39009 } }
I20260925 05:01:05.840719 3415 leader_election.cc:302] T 00000000000000000000000000000000 P a5c98d009c314e9087546177a733eb4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5c98d009c314e9087546177a733eb4b; no voters:
I20260925 05:01:05.841203 3415 leader_election.cc:288] T 00000000000000000000000000000000 P a5c98d009c314e9087546177a733eb4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:01:05.841336 3420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5c98d009c314e9087546177a733eb4b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:01:05.842001 3420 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5c98d009c314e9087546177a733eb4b [term 1 LEADER]: Becoming Leader. State: Replica: a5c98d009c314e9087546177a733eb4b, State: Running, Role: LEADER
I20260925 05:01:05.842450 3420 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5c98d009c314e9087546177a733eb4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5c98d009c314e9087546177a733eb4b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 39009 } }
I20260925 05:01:05.843153 3415 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5c98d009c314e9087546177a733eb4b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:01:05.845160 3422 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5c98d009c314e9087546177a733eb4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5c98d009c314e9087546177a733eb4b. Latest consensus state: current_term: 1 leader_uuid: "a5c98d009c314e9087546177a733eb4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5c98d009c314e9087546177a733eb4b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 39009 } } }
I20260925 05:01:05.845602 3422 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5c98d009c314e9087546177a733eb4b [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:05.845459 3421 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5c98d009c314e9087546177a733eb4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5c98d009c314e9087546177a733eb4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5c98d009c314e9087546177a733eb4b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 39009 } } }
I20260925 05:01:05.845762 3421 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5c98d009c314e9087546177a733eb4b [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:05.847308 3425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:01:05.851987 3425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:01:05.860103 3425 catalog_manager.cc:1347] Generated new cluster ID: c9732bb87e874ae791dc7df497dae895
I20260925 05:01:05.860224 3425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:01:05.871254 3425 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:01:05.872828 3425 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:01:05.883387 3425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5c98d009c314e9087546177a733eb4b: Generated new TSK 0
I20260925 05:01:05.884232 3425 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
I20260925 05:01:05.925383 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:0
--local_ip_for_outbound_sockets=127.30.247.1
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:39009
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:44107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:06.247788 3441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:06.248040 3441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:06.248131 3441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:06.255599 3441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:06.255704 3441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:06.255816 3441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:01:06.265189 3441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:44107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:39009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:06.266324 3441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:06.267541 3441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:06.280067 3446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:06.280100 3447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:06.280165 3449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:06.280862 3441 server_base.cc:1034] running on GCE node
I20260925 05:01:06.281461 3441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:06.282416 3441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:06.283638 3441 hybrid_clock.cc:648] HybridClock initialized: now 1790312466283594 us; error 38 us; skew 500 ppm
I20260925 05:01:06.283900 3441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:06 dist-test-slave-5n86 krb5kdc[3336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312466, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:06.286876 3441 init.cc:377] Logged in from keytab as kudu/127.30.247.1@KRBTEST.COM (short username kudu)
I20260925 05:01:06.288035 3441 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:06.288190 3441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:06.288504 3441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:06.291384 3441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-0/data/instance:
uuid: "3451976cd44d49329b9cd5d2fb78d3a8"
format_stamp: "Formatted at 2026-09-25 05:01:06 on dist-test-slave-5n86"
server_key: "42fa2dda9e410e4089fe76da4240b3d1"
server_key_iv: "e612a26601a86fae4b1db3c6d05da881"
server_key_version: "encryptionkey@0"
I20260925 05:01:06.292171 3441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance:
uuid: "3451976cd44d49329b9cd5d2fb78d3a8"
format_stamp: "Formatted at 2026-09-25 05:01:06 on dist-test-slave-5n86"
server_key: "42fa2dda9e410e4089fe76da4240b3d1"
server_key_iv: "e612a26601a86fae4b1db3c6d05da881"
server_key_version: "encryptionkey@0"
I20260925 05:01:06.296924 3441 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260925 05:01:06.300136 3455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:06.301566 3441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:01:06.301743 3441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "3451976cd44d49329b9cd5d2fb78d3a8"
format_stamp: "Formatted at 2026-09-25 05:01:06 on dist-test-slave-5n86"
server_key: "42fa2dda9e410e4089fe76da4240b3d1"
server_key_iv: "e612a26601a86fae4b1db3c6d05da881"
server_key_version: "encryptionkey@0"
I20260925 05:01:06.301956 3441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:06.318128 3441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:06.321821 3441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:06.322085 3441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:06.322952 3441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:06.324194 3441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:06.324318 3441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:06.324446 3441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:06.324539 3441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:06.358039 3441 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:41411
I20260925 05:01:06.358089 3567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:41411 every 8 connection(s)
I20260925 05:01:06.358883 3441 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:01:06.362411 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 3441
I20260925 05:01:06.362635 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance
I20260925 05:01:06.363314 31708 external_mini_cluster.cc:1428] Setting key 68d007f0b46b246aa3d45cf0686a99fb
Sep 25 05:01:06 dist-test-slave-5n86 krb5kdc[3336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312466, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:01:06.379981 3570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:06.361259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:37891 (local address 127.30.247.62:39009)
0925 05:01:06.361720 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:06.361725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:06.362633 (+ 908us) server_negotiation.cc:407] Connection header received
0925 05:01:06.364473 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:06.364481 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:06.364596 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:06.364735 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:06.366216 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:06.366935 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:06.368482 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:06.368954 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:06.369098 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:06.373148 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:06.373157 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:06.373165 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:06.373210 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:06.376716 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:06.377415 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:06.377423 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:06.377429 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:06.377573 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:06.377994 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:06.378001 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:06.378008 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:06.378433 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:06.378581 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:06.379260 (+ 679us) server_negotiation.cc:299] Negotiation successful
0925 05:01:06.379543 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":251,"threads_started":1}
I20260925 05:01:06.380779 3571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:06.361639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:39009 (local address 127.30.247.1:37891)
0925 05:01:06.362395 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:06.362434 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:06.364184 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:06.364964 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:06.364973 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:06.365479 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:06.366034 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:06.366054 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:06.367087 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:06.367093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:06.368328 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:06.368340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:06.369180 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:06.369189 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:06.369456 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:06.370133 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:06.370167 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:06.372923 (+ 2756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:06.376860 (+ 3937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:06.376867 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:06.376880 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:06.377288 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:06.377690 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:06.377693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:06.377695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:06.377880 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:06.378561 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:06.378566 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:06.378890 (+ 324us) client_negotiation.cc:769] Sending connection context
0925 05:01:06.379117 (+ 227us) client_negotiation.cc:240] Negotiation successful
0925 05:01:06.379604 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":340,"threads_started":1}
I20260925 05:01:06.382506 3568 heartbeater.cc:347] Connected to a master server at 127.30.247.62:39009
I20260925 05:01:06.382843 3568 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:06.383652 3568 heartbeater.cc:511] Master 127.30.247.62:39009 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.30.247.2@KRBTEST.COM; defaulting to no policy
I20260925 05:01:06.386083 3380 ts_manager.cc:194] Registered new tserver with Master: 3451976cd44d49329b9cd5d2fb78d3a8 (127.30.247.1:41411)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.2@KRBTEST.COM" created.
I20260925 05:01:06.388263 3380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.1@KRBTEST.COM'} at 127.30.247.1:37891
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
I20260925 05:01:06.412230 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:0
--local_ip_for_outbound_sockets=127.30.247.2
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:39009
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:44107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:06.721969 3574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:06.722221 3574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:06.722323 3574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:06.729373 3574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:06.729470 3574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:06.729580 3574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:01:06.738732 3574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:44107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:39009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:06.739845 3574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:06.741034 3574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:06.754169 3580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:06.754309 3579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:06.754206 3582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:06.758086 3574 server_base.cc:1034] running on GCE node
I20260925 05:01:06.758651 3574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:06.759543 3574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:06.760788 3574 hybrid_clock.cc:648] HybridClock initialized: now 1790312466760742 us; error 63 us; skew 500 ppm
I20260925 05:01:06.761121 3574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:06 dist-test-slave-5n86 krb5kdc[3336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312466, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:06.764137 3574 init.cc:377] Logged in from keytab as kudu/127.30.247.2@KRBTEST.COM (short username kudu)
I20260925 05:01:06.765311 3574 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:06.765398 3574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:06.765643 3574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:06.768396 3574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-1/data/instance:
uuid: "4ef517b2b98e4ed4889eb9fd433b3b74"
format_stamp: "Formatted at 2026-09-25 05:01:06 on dist-test-slave-5n86"
server_key: "c44d600499c4aca17bb6cc51e83d3435"
server_key_iv: "a3dfd0db4c2959596a066f0f1fcb1044"
server_key_version: "encryptionkey@0"
I20260925 05:01:06.769207 3574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance:
uuid: "4ef517b2b98e4ed4889eb9fd433b3b74"
format_stamp: "Formatted at 2026-09-25 05:01:06 on dist-test-slave-5n86"
server_key: "c44d600499c4aca17bb6cc51e83d3435"
server_key_iv: "a3dfd0db4c2959596a066f0f1fcb1044"
server_key_version: "encryptionkey@0"
I20260925 05:01:06.774058 3574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 05:01:06.777259 3588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:06.778606 3574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:01:06.778730 3574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "4ef517b2b98e4ed4889eb9fd433b3b74"
format_stamp: "Formatted at 2026-09-25 05:01:06 on dist-test-slave-5n86"
server_key: "c44d600499c4aca17bb6cc51e83d3435"
server_key_iv: "a3dfd0db4c2959596a066f0f1fcb1044"
server_key_version: "encryptionkey@0"
I20260925 05:01:06.778900 3574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:06.807677 3574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:06.814833 3574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:06.815126 3574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:06.816323 3574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:06.817747 3574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:06.817875 3574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:06.817989 3574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:06.818068 3574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:06.851899 3574 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:44879
I20260925 05:01:06.852020 3700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:44879 every 8 connection(s)
I20260925 05:01:06.852854 3574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:01:06.858999 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 3574
I20260925 05:01:06.859232 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance
I20260925 05:01:06.859695 31708 external_mini_cluster.cc:1428] Setting key ee674a2eb3ee868b519ce67bc2171e1f
Sep 25 05:01:06 dist-test-slave-5n86 krb5kdc[3336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312466, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:01:06.872588 3570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:06.855497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:58591 (local address 127.30.247.62:39009)
0925 05:01:06.855682 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:06.855686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:06.856927 (+ 1241us) server_negotiation.cc:407] Connection header received
0925 05:01:06.858721 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:06.858730 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:06.858843 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:06.858968 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:06.860868 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:06.861606 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:06.863159 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:06.863558 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:06.863698 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:06.867016 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:06.867026 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:06.867030 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:06.867060 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:06.869599 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:06.870298 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:06.870303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:06.870306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:06.870436 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:06.870861 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:06.870870 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:06.870872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:06.871122 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:06.871290 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:06.872010 (+ 720us) server_negotiation.cc:299] Negotiation successful
0925 05:01:06.872226 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 05:01:06.873569 3703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:06.855886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:39009 (local address 127.30.247.2:58591)
0925 05:01:06.856738 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:06.856787 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:06.858430 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:06.859274 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:06.859308 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:06.859939 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:06.860682 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:06.860703 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:06.861762 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:06.861767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:06.862996 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:06.863007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:06.863784 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:06.863791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:06.864014 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:06.864721 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:06.864747 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:06.866802 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:06.869740 (+ 2938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:06.869749 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:06.869765 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:06.870167 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:06.870550 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:06.870555 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:06.870559 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:06.870732 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:06.871254 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:06.871261 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:06.871621 (+ 360us) client_negotiation.cc:769] Sending connection context
0925 05:01:06.871926 (+ 305us) client_negotiation.cc:240] Negotiation successful
0925 05:01:06.872438 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":303,"threads_started":1}
I20260925 05:01:06.875145 3701 heartbeater.cc:347] Connected to a master server at 127.30.247.62:39009
I20260925 05:01:06.875481 3701 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:06.876296 3701 heartbeater.cc:511] Master 127.30.247.62:39009 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.30.247.3@KRBTEST.COM; defaulting to no policy
I20260925 05:01:06.877985 3380 ts_manager.cc:194] Registered new tserver with Master: 4ef517b2b98e4ed4889eb9fd433b3b74 (127.30.247.2:44879)
I20260925 05:01:06.878901 3380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.2@KRBTEST.COM'} at 127.30.247.2:58591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
I20260925 05:01:06.904771 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:0
--local_ip_for_outbound_sockets=127.30.247.3
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:39009
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:44107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:07.214004 3706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:07.214250 3706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:07.214416 3706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:07.221211 3706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:07.221310 3706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:07.221544 3706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:01:07.230607 3706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:44107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:39009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:07.231791 3706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:07.233037 3706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:07.245836 3714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:07.245932 3711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:07.245951 3712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:07.246601 3706 server_base.cc:1034] running on GCE node
I20260925 05:01:07.247125 3706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:07.248001 3706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:07.249194 3706 hybrid_clock.cc:648] HybridClock initialized: now 1790312467249152 us; error 35 us; skew 500 ppm
I20260925 05:01:07.249537 3706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:07 dist-test-slave-5n86 krb5kdc[3336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312467, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:07.256500 3706 init.cc:377] Logged in from keytab as kudu/127.30.247.3@KRBTEST.COM (short username kudu)
I20260925 05:01:07.257822 3706 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:07.257941 3706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:07.258277 3706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:07.261281 3706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-2/data/instance:
uuid: "5faf0eacceac424fb8e484383e83b77b"
format_stamp: "Formatted at 2026-09-25 05:01:07 on dist-test-slave-5n86"
server_key: "6013921c107662a225001e2bd5fa7efb"
server_key_iv: "3294d6e33a49c77d2e6c449a633c2ab9"
server_key_version: "encryptionkey@0"
I20260925 05:01:07.262069 3706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance:
uuid: "5faf0eacceac424fb8e484383e83b77b"
format_stamp: "Formatted at 2026-09-25 05:01:07 on dist-test-slave-5n86"
server_key: "6013921c107662a225001e2bd5fa7efb"
server_key_iv: "3294d6e33a49c77d2e6c449a633c2ab9"
server_key_version: "encryptionkey@0"
I20260925 05:01:07.267051 3706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:01:07.270387 3720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:07.271870 3706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:01:07.272038 3706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "5faf0eacceac424fb8e484383e83b77b"
format_stamp: "Formatted at 2026-09-25 05:01:07 on dist-test-slave-5n86"
server_key: "6013921c107662a225001e2bd5fa7efb"
server_key_iv: "3294d6e33a49c77d2e6c449a633c2ab9"
server_key_version: "encryptionkey@0"
I20260925 05:01:07.272257 3706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:07.289947 3706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:07.294082 3706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:07.294349 3706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:07.295259 3706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:07.296629 3706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:07.296813 3706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 05:01:07.296957 3706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:07.297039 3706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:07.330346 3706 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:40655
I20260925 05:01:07.330518 3832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:40655 every 8 connection(s)
I20260925 05:01:07.331175 3706 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:01:07.331540 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 3706
I20260925 05:01:07.331750 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance
I20260925 05:01:07.332324 31708 external_mini_cluster.cc:1428] Setting key 4a39b8363a5c48880f2a3401ffd054d1
Sep 25 05:01:07 dist-test-slave-5n86 krb5kdc[3336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312467, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:01:07.351274 3570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:07.333811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:53433 (local address 127.30.247.62:39009)
0925 05:01:07.333989 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:07.333994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:07.335515 (+ 1521us) server_negotiation.cc:407] Connection header received
0925 05:01:07.337337 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:07.337342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:07.337407 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:07.337492 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:07.338889 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.339608 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.341126 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.341521 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.341616 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:07.345134 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:07.345139 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:07.345142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:07.345174 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:07.348314 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:07.349213 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:07.349217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:07.349219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:07.349326 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:07.349736 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:07.349739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:07.349741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:07.349998 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:07.350101 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:07.350898 (+ 797us) server_negotiation.cc:299] Negotiation successful
0925 05:01:07.351043 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 05:01:07.352324 3835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:07.334566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:39009 (local address 127.30.247.3:53433)
0925 05:01:07.335371 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:07.335420 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:07.337084 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:07.337689 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:07.337697 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:07.338186 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:07.338733 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:07.338750 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.339735 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.339739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:07.341003 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:07.341014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.341696 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.341701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:07.341871 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:07.342558 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:07.342578 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:07.344947 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:07.348498 (+ 3551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:07.348509 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:07.348527 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:07.349097 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:07.349455 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:07.349459 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:07.349462 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:07.349633 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:07.350278 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:07.350283 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:07.350575 (+ 292us) client_negotiation.cc:769] Sending connection context
0925 05:01:07.350826 (+ 251us) client_negotiation.cc:240] Negotiation successful
0925 05:01:07.351175 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":337,"threads_started":1}
I20260925 05:01:07.353740 3833 heartbeater.cc:347] Connected to a master server at 127.30.247.62:39009
I20260925 05:01:07.354053 3833 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:07.354696 3833 heartbeater.cc:511] Master 127.30.247.62:39009 requested a full tablet report, sending...
I20260925 05:01:07.356067 3380 ts_manager.cc:194] Registered new tserver with Master: 5faf0eacceac424fb8e484383e83b77b (127.30.247.3:40655)
I20260925 05:01:07.356964 3380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.3@KRBTEST.COM'} at 127.30.247.3:53433
I20260925 05:01:07.359949 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:01:07.377192 3570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:07.364759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48116 (local address 127.30.247.62:39009)
0925 05:01:07.364924 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:07.364928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:07.365499 (+ 571us) server_negotiation.cc:407] Connection header received
0925 05:01:07.365603 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:07.365607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:07.365663 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:07.365747 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:07.367076 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.367785 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.369138 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.369571 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.369665 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:07.371724 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:07.371729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:07.371732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:07.371761 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:07.374488 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:07.375093 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:07.375097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:07.375099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:07.375220 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:07.375828 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:07.375832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:07.375834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:07.376167 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:07.376350 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:07.376828 (+ 478us) server_negotiation.cc:299] Negotiation successful
0925 05:01:07.376984 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 05:01:07.381268 3380 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:01:07.388207 3380 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:01:07.391456 3568 heartbeater.cc:503] Master 127.30.247.62:39009 was elected leader, sending a full tablet report...
I20260925 05:01:07.430138 3850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:07.401230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44906 (local address 127.30.247.1:41411)
0925 05:01:07.417760 (+ 16530us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:07.417774 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:07.420031 (+ 2257us) server_negotiation.cc:407] Connection header received
0925 05:01:07.420174 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:07.420186 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:07.420577 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:07.420818 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:01:07.422227 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.423933 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.427196 (+ 3263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.428653 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.428809 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:07.429029 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:07.429673 (+ 644us) server_negotiation.cc:299] Negotiation successful
0925 05:01:07.429836 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16279,"thread_start_us":16104,"threads_started":1}
I20260925 05:01:07.430147 3848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:07.400650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:40655 (local address 127.0.0.1:38692)
0925 05:01:07.420962 (+ 20312us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:07.420984 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:07.421110 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:07.421957 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:07.421962 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:07.421989 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:01:07.422423 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:07.422442 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.424345 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.424352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:07.427029 (+ 2677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:07.427038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.429197 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.429205 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:07.429377 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:07.429477 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 05:01:07.429560 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 05:01:07.429702 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20203,"mutex_wait_us":163,"thread_start_us":20069,"threads_started":1}
I20260925 05:01:07.430269 3846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:07.400517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.30.247.2:44879)
0925 05:01:07.422484 (+ 21967us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:07.422494 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:07.422522 (+ 28us) server_negotiation.cc:407] Connection header received
0925 05:01:07.422618 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:07.422623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:07.422785 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:07.422981 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:01:07.423495 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.424907 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.426857 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.428389 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.428528 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:07.428837 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:07.429542 (+ 705us) server_negotiation.cc:299] Negotiation successful
0925 05:01:07.429804 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21801,"thread_start_us":21628,"threads_started":1}
I20260925 05:01:07.430336 3849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:07.400915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38692 (local address 127.30.247.3:40655)
0925 05:01:07.420282 (+ 19367us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:07.420294 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:07.421259 (+ 965us) server_negotiation.cc:407] Connection header received
0925 05:01:07.421362 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:07.421369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:07.421614 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:07.421814 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:01:07.422658 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.424189 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.427194 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.428653 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.428809 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:07.429024 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:07.429838 (+ 814us) server_negotiation.cc:299] Negotiation successful
0925 05:01:07.430000 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19138,"thread_start_us":19385,"threads_started":1}
I20260925 05:01:07.431223 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:07.400632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:41411 (local address 127.0.0.1:44906)
0925 05:01:07.419693 (+ 19061us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:07.419717 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:07.419877 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:07.421444 (+ 1567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:07.421455 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:07.421498 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:01:07.421958 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:07.421974 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.424109 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.424118 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:07.427010 (+ 2892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:07.427029 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.429214 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.429220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:07.429384 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:07.429476 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 05:01:07.429561 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 05:01:07.430983 (+ 1422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18874,"mutex_wait_us":1532,"thread_start_us":20467,"threads_started":1}
I20260925 05:01:07.431864 3845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:07.400261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:44879 (local address 127.0.0.1:43262)
0925 05:01:07.415889 (+ 15628us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:07.415946 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:07.416155 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:07.422970 (+ 6815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:07.422974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:07.422998 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:01:07.423323 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:07.423332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.425062 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.425068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:07.426604 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:07.426616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.428965 (+ 2349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.428970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:07.429224 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:07.429289 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 05:01:07.429442 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 05:01:07.429642 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15295,"mutex_wait_us":1,"thread_start_us":15122,"threads_started":1}
I20260925 05:01:07.434584 3768 tablet_service.cc:1467] Processing CreateTablet for tablet cbf8e76594ac4a22bb2419cba6639aef (DEFAULT_TABLE table=test-table [id=10ee5672704840459b13a9a4b7afd9d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:01:07.434626 3503 tablet_service.cc:1467] Processing CreateTablet for tablet cbf8e76594ac4a22bb2419cba6639aef (DEFAULT_TABLE table=test-table [id=10ee5672704840459b13a9a4b7afd9d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:01:07.435160 3636 tablet_service.cc:1467] Processing CreateTablet for tablet cbf8e76594ac4a22bb2419cba6639aef (DEFAULT_TABLE table=test-table [id=10ee5672704840459b13a9a4b7afd9d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:01:07.436057 3768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbf8e76594ac4a22bb2419cba6639aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:07.436054 3503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbf8e76594ac4a22bb2419cba6639aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:07.436378 3636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbf8e76594ac4a22bb2419cba6639aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:07.449308 3851 tablet_bootstrap.cc:492] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74: Bootstrap starting.
I20260925 05:01:07.451285 3853 tablet_bootstrap.cc:492] T cbf8e76594ac4a22bb2419cba6639aef P 5faf0eacceac424fb8e484383e83b77b: Bootstrap starting.
I20260925 05:01:07.453048 3851 tablet_bootstrap.cc:654] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:07.453214 3852 tablet_bootstrap.cc:492] T cbf8e76594ac4a22bb2419cba6639aef P 3451976cd44d49329b9cd5d2fb78d3a8: Bootstrap starting.
I20260925 05:01:07.454639 3853 tablet_bootstrap.cc:654] T cbf8e76594ac4a22bb2419cba6639aef P 5faf0eacceac424fb8e484383e83b77b: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:07.454911 3851 log.cc:824] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74: Log is configured to *not* fsync() on all Append() calls
I20260925 05:01:07.455847 3853 log.cc:824] T cbf8e76594ac4a22bb2419cba6639aef P 5faf0eacceac424fb8e484383e83b77b: Log is configured to *not* fsync() on all Append() calls
I20260925 05:01:07.456480 3852 tablet_bootstrap.cc:654] T cbf8e76594ac4a22bb2419cba6639aef P 3451976cd44d49329b9cd5d2fb78d3a8: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:07.457556 3852 log.cc:824] T cbf8e76594ac4a22bb2419cba6639aef P 3451976cd44d49329b9cd5d2fb78d3a8: Log is configured to *not* fsync() on all Append() calls
I20260925 05:01:07.457964 3851 tablet_bootstrap.cc:492] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74: No bootstrap required, opened a new log
I20260925 05:01:07.458227 3851 ts_tablet_manager.cc:1397] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 05:01:07.458796 3853 tablet_bootstrap.cc:492] T cbf8e76594ac4a22bb2419cba6639aef P 5faf0eacceac424fb8e484383e83b77b: No bootstrap required, opened a new log
I20260925 05:01:07.459038 3853 ts_tablet_manager.cc:1397] T cbf8e76594ac4a22bb2419cba6639aef P 5faf0eacceac424fb8e484383e83b77b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260925 05:01:07.462901 3852 tablet_bootstrap.cc:492] T cbf8e76594ac4a22bb2419cba6639aef P 3451976cd44d49329b9cd5d2fb78d3a8: No bootstrap required, opened a new log
I20260925 05:01:07.463195 3852 ts_tablet_manager.cc:1397] T cbf8e76594ac4a22bb2419cba6639aef P 3451976cd44d49329b9cd5d2fb78d3a8: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.006s
I20260925 05:01:07.468734 3851 raft_consensus.cc:348] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef517b2b98e4ed4889eb9fd433b3b74" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44879 } } peers { permanent_uuid: "5faf0eacceac424fb8e484383e83b77b" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 40655 } } peers { permanent_uuid: "3451976cd44d49329b9cd5d2fb78d3a8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 41411 } }
I20260925 05:01:07.469165 3851 raft_consensus.cc:374] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:07.469280 3851 raft_consensus.cc:729] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ef517b2b98e4ed4889eb9fd433b3b74, State: Initialized, Role: FOLLOWER
I20260925 05:01:07.469290 3853 raft_consensus.cc:348] T cbf8e76594ac4a22bb2419cba6639aef P 5faf0eacceac424fb8e484383e83b77b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef517b2b98e4ed4889eb9fd433b3b74" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44879 } } peers { permanent_uuid: "5faf0eacceac424fb8e484383e83b77b" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 40655 } } peers { permanent_uuid: "3451976cd44d49329b9cd5d2fb78d3a8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 41411 } }
I20260925 05:01:07.469736 3853 raft_consensus.cc:374] T cbf8e76594ac4a22bb2419cba6639aef P 5faf0eacceac424fb8e484383e83b77b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:07.469861 3853 raft_consensus.cc:729] T cbf8e76594ac4a22bb2419cba6639aef P 5faf0eacceac424fb8e484383e83b77b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5faf0eacceac424fb8e484383e83b77b, State: Initialized, Role: FOLLOWER
I20260925 05:01:07.469908 3851 consensus_queue.cc:260] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef517b2b98e4ed4889eb9fd433b3b74" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44879 } } peers { permanent_uuid: "5faf0eacceac424fb8e484383e83b77b" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 40655 } } peers { permanent_uuid: "3451976cd44d49329b9cd5d2fb78d3a8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 41411 } }
I20260925 05:01:07.469854 3852 raft_consensus.cc:348] T cbf8e76594ac4a22bb2419cba6639aef P 3451976cd44d49329b9cd5d2fb78d3a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef517b2b98e4ed4889eb9fd433b3b74" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44879 } } peers { permanent_uuid: "5faf0eacceac424fb8e484383e83b77b" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 40655 } } peers { permanent_uuid: "3451976cd44d49329b9cd5d2fb78d3a8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 41411 } }
I20260925 05:01:07.470270 3852 raft_consensus.cc:374] T cbf8e76594ac4a22bb2419cba6639aef P 3451976cd44d49329b9cd5d2fb78d3a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:07.470392 3852 raft_consensus.cc:729] T cbf8e76594ac4a22bb2419cba6639aef P 3451976cd44d49329b9cd5d2fb78d3a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3451976cd44d49329b9cd5d2fb78d3a8, State: Initialized, Role: FOLLOWER
I20260925 05:01:07.470428 3853 consensus_queue.cc:260] T cbf8e76594ac4a22bb2419cba6639aef P 5faf0eacceac424fb8e484383e83b77b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef517b2b98e4ed4889eb9fd433b3b74" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44879 } } peers { permanent_uuid: "5faf0eacceac424fb8e484383e83b77b" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 40655 } } peers { permanent_uuid: "3451976cd44d49329b9cd5d2fb78d3a8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 41411 } }
I20260925 05:01:07.471046 3852 consensus_queue.cc:260] T cbf8e76594ac4a22bb2419cba6639aef P 3451976cd44d49329b9cd5d2fb78d3a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef517b2b98e4ed4889eb9fd433b3b74" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44879 } } peers { permanent_uuid: "5faf0eacceac424fb8e484383e83b77b" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 40655 } } peers { permanent_uuid: "3451976cd44d49329b9cd5d2fb78d3a8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 41411 } }
I20260925 05:01:07.471222 3701 heartbeater.cc:503] Master 127.30.247.62:39009 was elected leader, sending a full tablet report...
I20260925 05:01:07.472018 3851 ts_tablet_manager.cc:1428] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260925 05:01:07.473027 3833 heartbeater.cc:503] Master 127.30.247.62:39009 was elected leader, sending a full tablet report...
I20260925 05:01:07.474828 3853 ts_tablet_manager.cc:1428] T cbf8e76594ac4a22bb2419cba6639aef P 5faf0eacceac424fb8e484383e83b77b: Time spent starting tablet: real 0.016s user 0.014s sys 0.002s
I20260925 05:01:07.476611 3852 ts_tablet_manager.cc:1428] T cbf8e76594ac4a22bb2419cba6639aef P 3451976cd44d49329b9cd5d2fb78d3a8: Time spent starting tablet: real 0.013s user 0.002s sys 0.011s
W20260925 05:01:07.582893 3834 tablet.cc:2369] T cbf8e76594ac4a22bb2419cba6639aef P 5faf0eacceac424fb8e484383e83b77b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:01:07.604809 3702 tablet.cc:2369] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:01:07.611074 3569 tablet.cc:2369] T cbf8e76594ac4a22bb2419cba6639aef P 3451976cd44d49329b9cd5d2fb78d3a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:01:07.724959 3857 raft_consensus.cc:482] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:01:07.725224 3857 raft_consensus.cc:504] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef517b2b98e4ed4889eb9fd433b3b74" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44879 } } peers { permanent_uuid: "5faf0eacceac424fb8e484383e83b77b" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 40655 } } peers { permanent_uuid: "3451976cd44d49329b9cd5d2fb78d3a8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 41411 } }
I20260925 05:01:07.726835 3857 leader_election.cc:288] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5faf0eacceac424fb8e484383e83b77b (127.30.247.3:40655), 3451976cd44d49329b9cd5d2fb78d3a8 (127.30.247.1:41411)
I20260925 05:01:07.733055 3849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:07.727224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:49233 (local address 127.30.247.3:40655)
0925 05:01:07.727685 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:07.727694 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:07.727875 (+ 181us) server_negotiation.cc:407] Connection header received
0925 05:01:07.728105 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:07.728110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:07.728175 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:07.728277 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:01:07.728885 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.729619 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.731355 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.732106 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.732295 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:07.732383 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:07.732682 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 05:01:07.732802 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274}
I20260925 05:01:07.733155 3860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:07.727030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:40655 (local address 127.30.247.2:49233)
0925 05:01:07.727740 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:07.727768 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:07.727948 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:07.728293 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:07.728298 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:07.728321 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:01:07.728740 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:07.728750 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.729755 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.729762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:07.731182 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:07.731193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.732319 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.732325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:07.732496 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:07.732520 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 05:01:07.732657 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 05:01:07.732785 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":589,"threads_started":1}
I20260925 05:01:07.733301 3850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:07.727408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:45291 (local address 127.30.247.1:41411)
0925 05:01:07.727614 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:07.727622 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:07.728304 (+ 682us) server_negotiation.cc:407] Connection header received
0925 05:01:07.728503 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:07.728512 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:07.728629 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:07.728744 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:01:07.729266 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.729962 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.731650 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.732376 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.732552 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:07.732645 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:07.732941 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 05:01:07.733060 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 05:01:07.733773 3861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:07.727104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:41411 (local address 127.30.247.2:45291)
0925 05:01:07.728195 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:07.728209 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:07.728357 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:07.728756 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:07.728761 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:07.728782 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:01:07.729106 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:07.729114 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.730098 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.730104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:07.731492 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:07.731504 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.732617 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.732623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:07.732778 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:07.732799 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 05:01:07.732920 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 05:01:07.733036 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"thread_start_us":794,"threads_started":1}
I20260925 05:01:07.734254 3788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf8e76594ac4a22bb2419cba6639aef" candidate_uuid: "4ef517b2b98e4ed4889eb9fd433b3b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5faf0eacceac424fb8e484383e83b77b" is_pre_election: true
I20260925 05:01:07.734727 3788 raft_consensus.cc:2454] T cbf8e76594ac4a22bb2419cba6639aef P 5faf0eacceac424fb8e484383e83b77b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ef517b2b98e4ed4889eb9fd433b3b74 in term 0.
I20260925 05:01:07.734722 3523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf8e76594ac4a22bb2419cba6639aef" candidate_uuid: "4ef517b2b98e4ed4889eb9fd433b3b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3451976cd44d49329b9cd5d2fb78d3a8" is_pre_election: true
I20260925 05:01:07.735204 3523 raft_consensus.cc:2454] T cbf8e76594ac4a22bb2419cba6639aef P 3451976cd44d49329b9cd5d2fb78d3a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ef517b2b98e4ed4889eb9fd433b3b74 in term 0.
I20260925 05:01:07.735395 3589 leader_election.cc:302] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ef517b2b98e4ed4889eb9fd433b3b74, 5faf0eacceac424fb8e484383e83b77b; no voters:
I20260925 05:01:07.735774 3857 raft_consensus.cc:2781] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:01:07.735918 3857 raft_consensus.cc:482] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:01:07.736017 3857 raft_consensus.cc:3037] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:07.738129 3857 raft_consensus.cc:504] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef517b2b98e4ed4889eb9fd433b3b74" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44879 } } peers { permanent_uuid: "5faf0eacceac424fb8e484383e83b77b" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 40655 } } peers { permanent_uuid: "3451976cd44d49329b9cd5d2fb78d3a8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 41411 } }
I20260925 05:01:07.738718 3857 leader_election.cc:288] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74 [CANDIDATE]: Term 1 election: Requested vote from peers 5faf0eacceac424fb8e484383e83b77b (127.30.247.3:40655), 3451976cd44d49329b9cd5d2fb78d3a8 (127.30.247.1:41411)
I20260925 05:01:07.739070 3788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf8e76594ac4a22bb2419cba6639aef" candidate_uuid: "4ef517b2b98e4ed4889eb9fd433b3b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5faf0eacceac424fb8e484383e83b77b"
I20260925 05:01:07.739171 3523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf8e76594ac4a22bb2419cba6639aef" candidate_uuid: "4ef517b2b98e4ed4889eb9fd433b3b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3451976cd44d49329b9cd5d2fb78d3a8"
I20260925 05:01:07.739256 3788 raft_consensus.cc:3037] T cbf8e76594ac4a22bb2419cba6639aef P 5faf0eacceac424fb8e484383e83b77b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:07.739377 3523 raft_consensus.cc:3037] T cbf8e76594ac4a22bb2419cba6639aef P 3451976cd44d49329b9cd5d2fb78d3a8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:07.741454 3788 raft_consensus.cc:2454] T cbf8e76594ac4a22bb2419cba6639aef P 5faf0eacceac424fb8e484383e83b77b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ef517b2b98e4ed4889eb9fd433b3b74 in term 1.
I20260925 05:01:07.741525 3523 raft_consensus.cc:2454] T cbf8e76594ac4a22bb2419cba6639aef P 3451976cd44d49329b9cd5d2fb78d3a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ef517b2b98e4ed4889eb9fd433b3b74 in term 1.
I20260925 05:01:07.741792 3589 leader_election.cc:302] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ef517b2b98e4ed4889eb9fd433b3b74, 5faf0eacceac424fb8e484383e83b77b; no voters:
I20260925 05:01:07.742074 3857 raft_consensus.cc:2781] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:01:07.742513 3857 raft_consensus.cc:686] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74 [term 1 LEADER]: Becoming Leader. State: Replica: 4ef517b2b98e4ed4889eb9fd433b3b74, State: Running, Role: LEADER
I20260925 05:01:07.742991 3857 consensus_queue.cc:237] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef517b2b98e4ed4889eb9fd433b3b74" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44879 } } peers { permanent_uuid: "5faf0eacceac424fb8e484383e83b77b" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 40655 } } peers { permanent_uuid: "3451976cd44d49329b9cd5d2fb78d3a8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 41411 } }
I20260925 05:01:07.747924 3378 catalog_manager.cc:5495] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ef517b2b98e4ed4889eb9fd433b3b74 (127.30.247.2). New cstate: current_term: 1 leader_uuid: "4ef517b2b98e4ed4889eb9fd433b3b74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef517b2b98e4ed4889eb9fd433b3b74" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5faf0eacceac424fb8e484383e83b77b" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3451976cd44d49329b9cd5d2fb78d3a8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 41411 } health_report { overall_health: UNKNOWN } } }
I20260925 05:01:07.799806 3846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:07.794369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43274 (local address 127.30.247.2:44879)
0925 05:01:07.794604 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:07.794610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:07.794736 (+ 126us) server_negotiation.cc:407] Connection header received
0925 05:01:07.794992 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:07.794996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:07.795090 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:07.795274 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:01:07.795636 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.796355 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.797716 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.798111 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.798263 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:07.798737 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:01:07.799228 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:01:07.799403 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:07.799454 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 05:01:07.799532 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 05:01:07.808372 3523 raft_consensus.cc:1261] T cbf8e76594ac4a22bb2419cba6639aef P 3451976cd44d49329b9cd5d2fb78d3a8 [term 1 FOLLOWER]: Refusing update from remote peer 4ef517b2b98e4ed4889eb9fd433b3b74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:01:07.808437 3788 raft_consensus.cc:1261] T cbf8e76594ac4a22bb2419cba6639aef P 5faf0eacceac424fb8e484383e83b77b [term 1 FOLLOWER]: Refusing update from remote peer 4ef517b2b98e4ed4889eb9fd433b3b74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:01:07.809424 3862 consensus_queue.cc:1035] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3451976cd44d49329b9cd5d2fb78d3a8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 41411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:01:07.809796 3857 consensus_queue.cc:1035] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5faf0eacceac424fb8e484383e83b77b" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:01:07.828361 3865 mvcc.cc:205] Tried to move back new op lower bound from 7333119868132958208 to 7333119867879342080. Current Snapshot: MvccSnapshot[applied={T|T < 7333119868132958208}]
I20260925 05:01:07.830016 3866 mvcc.cc:205] Tried to move back new op lower bound from 7333119868132958208 to 7333119867879342080. Current Snapshot: MvccSnapshot[applied={T|T < 7333119868132958208}]
I20260925 05:01:07.850673 3378 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:01:07.850976 3378 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:01:07.854641 3378 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5c98d009c314e9087546177a733eb4b: Sending DeleteTablet for 3 replicas of tablet cbf8e76594ac4a22bb2419cba6639aef
I20260925 05:01:07.855757 3768 tablet_service.cc:1514] Processing DeleteTablet for tablet cbf8e76594ac4a22bb2419cba6639aef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:01:07 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:38692
I20260925 05:01:07.855782 3636 tablet_service.cc:1514] Processing DeleteTablet for tablet cbf8e76594ac4a22bb2419cba6639aef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:01:07 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:43262
I20260925 05:01:07.857568 3503 tablet_service.cc:1514] Processing DeleteTablet for tablet cbf8e76594ac4a22bb2419cba6639aef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:01:07 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:44906
I20260925 05:01:07.867304 3877 tablet_replica.cc:330] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74: stopping tablet replica
I20260925 05:01:07.867442 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 3441
I20260925 05:01:07.867878 3877 raft_consensus.cc:2229] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:01:07.868618 3877 raft_consensus.cc:2258] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:01:07.869181 3876 tablet_replica.cc:330] T cbf8e76594ac4a22bb2419cba6639aef P 5faf0eacceac424fb8e484383e83b77b: stopping tablet replica
I20260925 05:01:07.869809 3876 raft_consensus.cc:2229] T cbf8e76594ac4a22bb2419cba6639aef P 5faf0eacceac424fb8e484383e83b77b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:01:07.871795 3876 raft_consensus.cc:2258] T cbf8e76594ac4a22bb2419cba6639aef P 5faf0eacceac424fb8e484383e83b77b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:01:07.873019 3877 ts_tablet_manager.cc:1905] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:01:07.877676 3876 ts_tablet_manager.cc:1905] T cbf8e76594ac4a22bb2419cba6639aef P 5faf0eacceac424fb8e484383e83b77b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:01:07.886847 3876 ts_tablet_manager.cc:1918] T cbf8e76594ac4a22bb2419cba6639aef P 5faf0eacceac424fb8e484383e83b77b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:01:07.887059 3876 log.cc:1196] T cbf8e76594ac4a22bb2419cba6639aef P 5faf0eacceac424fb8e484383e83b77b: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-2/wal/wals/cbf8e76594ac4a22bb2419cba6639aef
I20260925 05:01:07.887570 3876 ts_tablet_manager.cc:1939] T cbf8e76594ac4a22bb2419cba6639aef P 5faf0eacceac424fb8e484383e83b77b: Deleting consensus metadata
I20260925 05:01:07.887795 3877 ts_tablet_manager.cc:1918] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:01:07.887991 3877 log.cc:1196] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-1/wal/wals/cbf8e76594ac4a22bb2419cba6639aef
I20260925 05:01:07.888674 3877 ts_tablet_manager.cc:1939] T cbf8e76594ac4a22bb2419cba6639aef P 4ef517b2b98e4ed4889eb9fd433b3b74: Deleting consensus metadata
I20260925 05:01:07.889398 3878 tablet_replica.cc:330] T cbf8e76594ac4a22bb2419cba6639aef P 3451976cd44d49329b9cd5d2fb78d3a8: stopping tablet replica
I20260925 05:01:07.889473 3364 catalog_manager.cc:4841] TS 5faf0eacceac424fb8e484383e83b77b (127.30.247.3:40655): tablet cbf8e76594ac4a22bb2419cba6639aef (table test-table [id=10ee5672704840459b13a9a4b7afd9d3]) successfully deleted
I20260925 05:01:07.890051 3878 raft_consensus.cc:2229] T cbf8e76594ac4a22bb2419cba6639aef P 3451976cd44d49329b9cd5d2fb78d3a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:01:07.890394 3367 catalog_manager.cc:4841] TS 4ef517b2b98e4ed4889eb9fd433b3b74 (127.30.247.2:44879): tablet cbf8e76594ac4a22bb2419cba6639aef (table test-table [id=10ee5672704840459b13a9a4b7afd9d3]) successfully deleted
I20260925 05:01:07.891122 3878 raft_consensus.cc:2258] T cbf8e76594ac4a22bb2419cba6639aef P 3451976cd44d49329b9cd5d2fb78d3a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:01:07.893392 3878 ts_tablet_manager.cc:1905] T cbf8e76594ac4a22bb2419cba6639aef P 3451976cd44d49329b9cd5d2fb78d3a8: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 05:01:07 dist-test-slave-5n86 krb5kdc[3336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.1@KRBTEST.COM
I20260925 05:01:07.900418 3878 ts_tablet_manager.cc:1918] T cbf8e76594ac4a22bb2419cba6639aef P 3451976cd44d49329b9cd5d2fb78d3a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:01:07.900614 3878 log.cc:1196] T cbf8e76594ac4a22bb2419cba6639aef P 3451976cd44d49329b9cd5d2fb78d3a8: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312430481417-31708-0/minicluster-data/ts-0/wal/wals/cbf8e76594ac4a22bb2419cba6639aef
I20260925 05:01:07.901113 3878 ts_tablet_manager.cc:1939] T cbf8e76594ac4a22bb2419cba6639aef P 3451976cd44d49329b9cd5d2fb78d3a8: Deleting consensus metadata
I20260925 05:01:07.902848 3365 catalog_manager.cc:4841] TS 3451976cd44d49329b9cd5d2fb78d3a8 (127.30.247.1:41411): tablet cbf8e76594ac4a22bb2419cba6639aef (table test-table [id=10ee5672704840459b13a9a4b7afd9d3]) successfully deleted
I20260925 05:01:07.904024 3850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:07.869724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44914 (local address 127.30.247.1:41411)
0925 05:01:07.869927 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:07.869933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:07.886542 (+ 16609us) server_negotiation.cc:407] Connection header received
0925 05:01:07.887076 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:07.887083 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:07.887171 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:07.887239 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:07.890716 (+ 3477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.891788 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.893266 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:07.893780 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:07.893945 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:07.897851 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:07.897861 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:07.897875 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:07.897908 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:07.901003 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:07.901688 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:07.901695 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:07.901697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:07.901845 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:07.902246 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:07.902257 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:07.902261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:07.902630 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:07.902898 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:07.903431 (+ 533us) server_negotiation.cc:299] Negotiation successful
0925 05:01:07.903647 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 05:01:07.904704 3563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:08.007148 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 3441
I20260925 05:01:08.034650 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 3574
Sep 25 05:01:08 dist-test-slave-5n86 krb5kdc[3336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.2@KRBTEST.COM
I20260925 05:01:08.061753 3846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:08.035586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43284 (local address 127.30.247.2:44879)
0925 05:01:08.036201 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:08.036206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:08.036225 (+ 19us) server_negotiation.cc:407] Connection header received
0925 05:01:08.036293 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:08.036298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:08.036402 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:08.036501 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:08.038336 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:08.039560 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:08.041796 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:08.042467 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:08.042608 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:08.052081 (+ 9473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:08.052107 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:08.052133 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:08.052202 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:08.057922 (+ 5720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:08.058807 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:08.058822 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:08.058828 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:08.059048 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:08.059543 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:08.059551 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:08.059557 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:08.060146 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:08.060619 (+ 473us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:08.060849 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 05:01:08.061151 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472}
I20260925 05:01:08.062947 3696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:08.176676 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 3574
I20260925 05:01:08.194607 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 3706
Sep 25 05:01:08 dist-test-slave-5n86 krb5kdc[3336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.3@KRBTEST.COM
I20260925 05:01:08.209419 3849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:08.195537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38694 (local address 127.30.247.3:40655)
0925 05:01:08.195726 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:08.195731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:08.195752 (+ 21us) server_negotiation.cc:407] Connection header received
0925 05:01:08.195932 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:08.195936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:08.196001 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:08.196084 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:08.197412 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:08.198131 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:08.199361 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:08.199775 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:08.199883 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:08.204053 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:08.204067 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:08.204083 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:08.204118 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:08.206876 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:08.207474 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:08.207481 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:08.207484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:08.207617 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:08.207955 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:08.207958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:08.207960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:08.208304 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:08.208516 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:08.208897 (+ 381us) server_negotiation.cc:299] Negotiation successful
0925 05:01:08.209055 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 05:01:08.210053 3828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:08.304695 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 3706
I20260925 05:01:08.320613 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 3349
I20260925 05:01:08.321738 3410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:08.428149 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 3349
2026-09-25T05:01:08Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4286 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T05:01:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:01:08Z Disabled control of system clock
I20260925 05:01:08.492826 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:45863
--webserver_interface=127.30.247.62
--webserver_port=0
--builtin_ntp_servers=127.30.247.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:45863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 05:01:08.799594 3887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:08.799853 3887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:08.799993 3887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:08.806969 3887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:01:08.807060 3887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:08.807129 3887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:08.807209 3887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:01:08.807282 3887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:01:08.816037 3887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:45863
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:45863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:08.817312 3887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:08.818979 3887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:08.830960 3892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:08.830989 3893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:08.831029 3895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:08.831584 3887 server_base.cc:1034] running on GCE node
I20260925 05:01:08.832299 3887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:08.833700 3887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:08.834945 3887 hybrid_clock.cc:648] HybridClock initialized: now 1790312468834932 us; error 58 us; skew 500 ppm
I20260925 05:01:08.835281 3887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:01:08.837692 3887 webserver.cc:460] Webserver started at http://127.30.247.62:34907/ using document root <none> and password file <none>
I20260925 05:01:08.838361 3887 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:08.838492 3887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:08.838810 3887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:08.841617 3887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "d9f20712d9824b3f8618b8dd75d81b9e"
format_stamp: "Formatted at 2026-09-25 05:01:08 on dist-test-slave-5n86"
server_key: "81ccf2aece2c146b3463e1c3edb5ae8d"
server_key_iv: "13930d79042ffbb2ac7d11835f5ad8d4"
server_key_version: "encryptionkey@0"
I20260925 05:01:08.842382 3887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "d9f20712d9824b3f8618b8dd75d81b9e"
format_stamp: "Formatted at 2026-09-25 05:01:08 on dist-test-slave-5n86"
server_key: "81ccf2aece2c146b3463e1c3edb5ae8d"
server_key_iv: "13930d79042ffbb2ac7d11835f5ad8d4"
server_key_version: "encryptionkey@0"
I20260925 05:01:08.847244 3887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 05:01:08.850662 3901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:08.851933 3887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:01:08.852114 3887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "d9f20712d9824b3f8618b8dd75d81b9e"
format_stamp: "Formatted at 2026-09-25 05:01:08 on dist-test-slave-5n86"
server_key: "81ccf2aece2c146b3463e1c3edb5ae8d"
server_key_iv: "13930d79042ffbb2ac7d11835f5ad8d4"
server_key_version: "encryptionkey@0"
I20260925 05:01:08.852330 3887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:08.858199 3887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:08.858994 3887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:08.859289 3887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:08.877789 3887 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:45863
I20260925 05:01:08.877888 3952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:45863 every 8 connection(s)
I20260925 05:01:08.879164 3887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:01:08.882856 3953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:08.888607 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 3887
I20260925 05:01:08.888891 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:01:08.889593 31708 external_mini_cluster.cc:1428] Setting key abe6d884e4063e411e49cbe9c79f84a7
I20260925 05:01:08.893965 3953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e: Bootstrap starting.
I20260925 05:01:08.897204 3953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:08.897924 3956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:08.891644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48912 (local address 127.30.247.62:45863)
0925 05:01:08.892670 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:08.892681 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:08.892731 (+ 50us) server_negotiation.cc:407] Connection header received
0925 05:01:08.894371 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:08.894393 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:08.894789 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:08.895163 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:08.896231 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:08.896236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:08.896249 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:01:08.896261 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:08.896315 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:01:08.896329 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:08.896525 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:08.896691 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 05:01:08.896938 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":309,"threads_started":1}
I20260925 05:01:08.898348 3953 log.cc:824] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e: Log is configured to *not* fsync() on all Append() calls
I20260925 05:01:08.901178 3953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e: No bootstrap required, opened a new log
I20260925 05:01:08.908072 3953 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f20712d9824b3f8618b8dd75d81b9e" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45863 } }
I20260925 05:01:08.908363 3953 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:08.908424 3953 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9f20712d9824b3f8618b8dd75d81b9e, State: Initialized, Role: FOLLOWER
I20260925 05:01:08.909139 3953 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f20712d9824b3f8618b8dd75d81b9e" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45863 } }
I20260925 05:01:08.909422 3958 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:01:08.909657 3958 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f20712d9824b3f8618b8dd75d81b9e" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45863 } }
I20260925 05:01:08.909438 3953 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:01:08.910306 3953 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:01:08.910252 3958 leader_election.cc:302] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9f20712d9824b3f8618b8dd75d81b9e; no voters:
I20260925 05:01:08.910405 3953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:08.911031 3958 leader_election.cc:288] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260925 05:01:08.912938 3953 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f20712d9824b3f8618b8dd75d81b9e" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45863 } }
I20260925 05:01:08.913159 3953 leader_election.cc:302] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9f20712d9824b3f8618b8dd75d81b9e; no voters:
I20260925 05:01:08.913203 3959 raft_consensus.cc:2741] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260925 05:01:08.913376 3953 leader_election.cc:288] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:01:08.913486 3958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:01:08.915088 3953 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:01:08.916370 3958 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e [term 1 LEADER]: Becoming Leader. State: Replica: d9f20712d9824b3f8618b8dd75d81b9e, State: Running, Role: LEADER
I20260925 05:01:08.917049 3958 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f20712d9824b3f8618b8dd75d81b9e" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45863 } }
I20260925 05:01:08.919041 3960 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9f20712d9824b3f8618b8dd75d81b9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f20712d9824b3f8618b8dd75d81b9e" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45863 } } }
I20260925 05:01:08.919469 3960 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:08.919981 3959 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9f20712d9824b3f8618b8dd75d81b9e. Latest consensus state: current_term: 1 leader_uuid: "d9f20712d9824b3f8618b8dd75d81b9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f20712d9824b3f8618b8dd75d81b9e" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45863 } } }
I20260925 05:01:08.920331 3959 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:08.921782 3967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:01:08.925904 3967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:01:08.933629 3967 catalog_manager.cc:1347] Generated new cluster ID: 9da3111db6a54645baa000c4e0702aa8
I20260925 05:01:08.933724 3967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:01:08.950793 3967 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:01:08.951165 3967 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:01:08.960057 3967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e: Generated new TSK 0
I20260925 05:01:08.960886 3967 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:01:08.967355 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:0
--local_ip_for_outbound_sockets=127.30.247.1
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:45863
--builtin_ntp_servers=127.30.247.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 05:01:09.302627 3978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:09.302879 3978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:09.302976 3978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:09.309999 3978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:09.310091 3978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:09.310195 3978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:01:09.319079 3978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:45863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:09.320218 3978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:09.321951 3978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:09.334445 3983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:09.334477 3986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:09.334501 3984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:09.335527 3978 server_base.cc:1034] running on GCE node
I20260925 05:01:09.336165 3978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:09.337105 3978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:09.338351 3978 hybrid_clock.cc:648] HybridClock initialized: now 1790312469338307 us; error 57 us; skew 500 ppm
I20260925 05:01:09.338660 3978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:01:09.340785 3978 webserver.cc:460] Webserver started at http://127.30.247.1:36141/ using document root <none> and password file <none>
I20260925 05:01:09.341435 3978 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:09.341573 3978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:09.341907 3978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:09.344511 3978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-0/data/instance:
uuid: "0453504661a240be9f932b749bdd7a12"
format_stamp: "Formatted at 2026-09-25 05:01:09 on dist-test-slave-5n86"
server_key: "eb8c37f83920bcb2d13590e7f02cfa85"
server_key_iv: "3319da6cbe6a1abc4c5dd93d0d79da80"
server_key_version: "encryptionkey@0"
I20260925 05:01:09.345347 3978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance:
uuid: "0453504661a240be9f932b749bdd7a12"
format_stamp: "Formatted at 2026-09-25 05:01:09 on dist-test-slave-5n86"
server_key: "eb8c37f83920bcb2d13590e7f02cfa85"
server_key_iv: "3319da6cbe6a1abc4c5dd93d0d79da80"
server_key_version: "encryptionkey@0"
I20260925 05:01:09.350219 3978 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 05:01:09.353392 3992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:09.354601 3978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:01:09.354780 3978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "0453504661a240be9f932b749bdd7a12"
format_stamp: "Formatted at 2026-09-25 05:01:09 on dist-test-slave-5n86"
server_key: "eb8c37f83920bcb2d13590e7f02cfa85"
server_key_iv: "3319da6cbe6a1abc4c5dd93d0d79da80"
server_key_version: "encryptionkey@0"
I20260925 05:01:09.355016 3978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:09.360525 3978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:09.361339 3978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:09.361618 3978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:09.362643 3978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:09.363937 3978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:09.364017 3978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:09.364085 3978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:09.364138 3978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:09.395828 3978 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:41399
I20260925 05:01:09.395887 4104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:41399 every 8 connection(s)
I20260925 05:01:09.397126 3978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:01:09.398922 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 3978
I20260925 05:01:09.399130 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance
I20260925 05:01:09.399948 31708 external_mini_cluster.cc:1428] Setting key c1a61dd2130a9698fb1fbacdda06d0af
I20260925 05:01:09.402663 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:0
--local_ip_for_outbound_sockets=127.30.247.2
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:45863
--builtin_ntp_servers=127.30.247.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 05:01:09.405720 4107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:09.399657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:54101 (local address 127.30.247.62:45863)
0925 05:01:09.400152 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:09.400167 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:09.401634 (+ 1467us) server_negotiation.cc:407] Connection header received
0925 05:01:09.403369 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:09.403378 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:09.403445 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:09.403567 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:09.404889 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:09.404897 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:09.404905 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:01:09.404928 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:09.404981 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:01:09.404998 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:09.405123 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:09.405297 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 05:01:09.405360 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":243,"threads_started":1}
I20260925 05:01:09.406461 4108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:09.400397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:45863 (local address 127.30.247.1:54101)
0925 05:01:09.401478 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:09.401520 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:09.403109 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:09.403744 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:09.403751 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:09.403837 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:09.404512 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:01:09.404515 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:09.404616 (+ 101us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:01:09.404712 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:09.405099 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:09.405106 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:09.405111 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:01:09.405318 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 05:01:09.405522 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":337,"threads_started":1}
I20260925 05:01:09.408144 4105 heartbeater.cc:347] Connected to a master server at 127.30.247.62:45863
I20260925 05:01:09.408486 4105 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:09.409176 4105 heartbeater.cc:511] Master 127.30.247.62:45863 requested a full tablet report, sending...
I20260925 05:01:09.411278 3918 ts_manager.cc:194] Registered new tserver with Master: 0453504661a240be9f932b749bdd7a12 (127.30.247.1:41399)
W20260925 05:01:09.714933 4109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:09.715234 4109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:09.715374 4109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:09.722445 4109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:09.722577 4109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:09.722748 4109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:01:09.731812 4109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:45863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:09.733068 4109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:09.734709 4109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:09.747335 4114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:09.747565 4115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:09.747640 4117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:09.748242 4109 server_base.cc:1034] running on GCE node
I20260925 05:01:09.748914 4109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:09.749756 4109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:09.750955 4109 hybrid_clock.cc:648] HybridClock initialized: now 1790312469750910 us; error 40 us; skew 500 ppm
I20260925 05:01:09.751266 4109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:01:09.753449 4109 webserver.cc:460] Webserver started at http://127.30.247.2:34557/ using document root <none> and password file <none>
I20260925 05:01:09.754112 4109 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:09.754195 4109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:09.754544 4109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:09.757403 4109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-1/data/instance:
uuid: "8f570660fe0f4a8c8e5d694c6bf7cefd"
format_stamp: "Formatted at 2026-09-25 05:01:09 on dist-test-slave-5n86"
server_key: "9ad9c7af125538f1a73b50a96864711b"
server_key_iv: "2292839c8d70357100101b8929d66f38"
server_key_version: "encryptionkey@0"
I20260925 05:01:09.758167 4109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance:
uuid: "8f570660fe0f4a8c8e5d694c6bf7cefd"
format_stamp: "Formatted at 2026-09-25 05:01:09 on dist-test-slave-5n86"
server_key: "9ad9c7af125538f1a73b50a96864711b"
server_key_iv: "2292839c8d70357100101b8929d66f38"
server_key_version: "encryptionkey@0"
I20260925 05:01:09.763248 4109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:01:09.766423 4123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:09.767701 4109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:01:09.767872 4109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "8f570660fe0f4a8c8e5d694c6bf7cefd"
format_stamp: "Formatted at 2026-09-25 05:01:09 on dist-test-slave-5n86"
server_key: "9ad9c7af125538f1a73b50a96864711b"
server_key_iv: "2292839c8d70357100101b8929d66f38"
server_key_version: "encryptionkey@0"
I20260925 05:01:09.768067 4109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:09.773644 4109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:09.774372 4109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:09.774636 4109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:09.775522 4109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:09.776883 4109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:09.776966 4109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:09.777093 4109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:09.777191 4109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:09.809695 4109 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:45705
I20260925 05:01:09.809830 4235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:45705 every 8 connection(s)
I20260925 05:01:09.810981 4109 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:01:09.811502 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 4109
I20260925 05:01:09.811692 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance
I20260925 05:01:09.812372 31708 external_mini_cluster.cc:1428] Setting key b0f3ed85387f12db8d117a83424e5b31
I20260925 05:01:09.815496 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:0
--local_ip_for_outbound_sockets=127.30.247.3
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:45863
--builtin_ntp_servers=127.30.247.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 05:01:09.819581 4107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:09.813679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:46449 (local address 127.30.247.62:45863)
0925 05:01:09.813880 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:09.813887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:09.815457 (+ 1570us) server_negotiation.cc:407] Connection header received
0925 05:01:09.817273 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:09.817282 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:09.817362 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:09.817482 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:09.818756 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:09.818765 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:09.818773 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:01:09.818797 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:09.818840 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:01:09.818860 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:09.819005 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:09.819159 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 05:01:09.819226 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 05:01:09.820312 4238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:09.814235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:45863 (local address 127.30.247.2:46449)
0925 05:01:09.815302 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:09.815344 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:09.817007 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:09.817655 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:09.817662 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:09.817745 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:09.818456 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:01:09.818459 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:09.818494 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:01:09.818572 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:09.818965 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:09.818971 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:09.818977 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 05:01:09.819170 (+ 193us) client_negotiation.cc:240] Negotiation successful
0925 05:01:09.819348 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":336,"threads_started":1}
I20260925 05:01:09.821668 4236 heartbeater.cc:347] Connected to a master server at 127.30.247.62:45863
I20260925 05:01:09.821964 4236 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:09.822494 4236 heartbeater.cc:511] Master 127.30.247.62:45863 requested a full tablet report, sending...
I20260925 05:01:09.823853 3918 ts_manager.cc:194] Registered new tserver with Master: 8f570660fe0f4a8c8e5d694c6bf7cefd (127.30.247.2:45705)
W20260925 05:01:10.139492 4239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:10.139743 4239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:10.139827 4239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:10.148066 4239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:10.148160 4239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:10.148262 4239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:01:10.157809 4239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:45863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:10.159101 4239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:10.160822 4239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:10.173430 4244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:10.173565 4245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:10.173512 4247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:10.174817 4239 server_base.cc:1034] running on GCE node
I20260925 05:01:10.175429 4239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:10.176307 4239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:10.177497 4239 hybrid_clock.cc:648] HybridClock initialized: now 1790312470177443 us; error 51 us; skew 500 ppm
I20260925 05:01:10.177836 4239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:01:10.179955 4239 webserver.cc:460] Webserver started at http://127.30.247.3:36107/ using document root <none> and password file <none>
I20260925 05:01:10.180665 4239 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:10.180753 4239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:10.181061 4239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:10.183643 4239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-2/data/instance:
uuid: "a6b5ee26c88d452880a02db12f312823"
format_stamp: "Formatted at 2026-09-25 05:01:10 on dist-test-slave-5n86"
server_key: "cd614d121b81dfefec5f216e3e99a3e6"
server_key_iv: "63d52a517beeac8113977cf009a068a0"
server_key_version: "encryptionkey@0"
I20260925 05:01:10.184405 4239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance:
uuid: "a6b5ee26c88d452880a02db12f312823"
format_stamp: "Formatted at 2026-09-25 05:01:10 on dist-test-slave-5n86"
server_key: "cd614d121b81dfefec5f216e3e99a3e6"
server_key_iv: "63d52a517beeac8113977cf009a068a0"
server_key_version: "encryptionkey@0"
I20260925 05:01:10.189179 4239 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 05:01:10.192415 4253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:10.193678 4239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:01:10.193847 4239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "a6b5ee26c88d452880a02db12f312823"
format_stamp: "Formatted at 2026-09-25 05:01:10 on dist-test-slave-5n86"
server_key: "cd614d121b81dfefec5f216e3e99a3e6"
server_key_iv: "63d52a517beeac8113977cf009a068a0"
server_key_version: "encryptionkey@0"
I20260925 05:01:10.194077 4239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:10.199359 4239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:10.200091 4239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:10.200357 4239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:10.201303 4239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:10.202591 4239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:10.202713 4239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:10.202841 4239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:10.202944 4239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:10.236048 4239 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:45055
I20260925 05:01:10.236105 4365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:45055 every 8 connection(s)
I20260925 05:01:10.237461 4239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:01:10.245126 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 4239
I20260925 05:01:10.245319 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance
I20260925 05:01:10.245771 31708 external_mini_cluster.cc:1428] Setting key e74b673831abf5c5c6750b4414b389cc
I20260925 05:01:10.245990 4107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:10.240102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:53411 (local address 127.30.247.62:45863)
0925 05:01:10.240317 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:10.240324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:10.241855 (+ 1531us) server_negotiation.cc:407] Connection header received
0925 05:01:10.243467 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:10.243472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:10.243524 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:10.243611 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:10.245114 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:10.245119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:10.245122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:01:10.245138 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:10.245170 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:01:10.245182 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:10.245423 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:10.245648 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 05:01:10.245718 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 05:01:10.247275 4368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:10.240774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:45863 (local address 127.30.247.3:53411)
0925 05:01:10.241687 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:10.241726 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:10.243218 (+ 1492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:10.243819 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:10.243848 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:10.243941 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:10.244689 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:01:10.244694 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:10.244751 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:01:10.244886 (+ 135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:10.245390 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:10.245401 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:10.245410 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 05:01:10.245732 (+ 322us) client_negotiation.cc:240] Negotiation successful
0925 05:01:10.246030 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":262,"threads_started":1}
I20260925 05:01:10.248807 4366 heartbeater.cc:347] Connected to a master server at 127.30.247.62:45863
I20260925 05:01:10.249148 4366 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:10.249775 4366 heartbeater.cc:511] Master 127.30.247.62:45863 requested a full tablet report, sending...
I20260925 05:01:10.251191 3918 ts_manager.cc:194] Registered new tserver with Master: a6b5ee26c88d452880a02db12f312823 (127.30.247.3:45055)
I20260925 05:01:10.261241 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:01:10.267970 4107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:10.265731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44376 (local address 127.30.247.62:45863)
0925 05:01:10.265938 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:10.265946 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:10.266446 (+ 500us) server_negotiation.cc:407] Connection header received
0925 05:01:10.266569 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:10.266573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:10.266623 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:10.266703 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:10.267579 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:10.267583 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:10.267586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:01:10.267597 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:10.267621 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:01:10.267632 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:10.267717 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:10.267752 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 05:01:10.267790 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 05:01:10.271811 3918 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44376:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:01:10.278396 3918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:01:10.296155 4379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:10.289474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:41399 (local address 127.0.0.1:55156)
0925 05:01:10.290297 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:10.290339 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:10.290469 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:10.292566 (+ 2097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:10.292574 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:10.292660 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:10.293437 (+ 777us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:01:10.293441 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:10.293469 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:01:10.293558 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:10.295390 (+ 1832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:10.295399 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:10.295406 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 05:01:10.295552 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 05:01:10.295642 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":2310,"threads_started":1}
I20260925 05:01:10.296873 4381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:10.290147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57610 (local address 127.30.247.2:45705)
0925 05:01:10.290946 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:10.290954 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:10.292318 (+ 1364us) server_negotiation.cc:407] Connection header received
0925 05:01:10.292390 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:10.292403 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:10.292638 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:10.292782 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:10.295416 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:10.295424 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:10.295436 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:01:10.295462 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:10.295531 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:01:10.295553 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:10.295703 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:10.296377 (+ 674us) server_negotiation.cc:299] Negotiation successful
0925 05:01:10.296474 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":814,"threads_started":1}
I20260925 05:01:10.297396 4378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:10.289582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:45705 (local address 127.0.0.1:57610)
0925 05:01:10.292075 (+ 2493us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:10.292091 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:10.292186 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:10.294504 (+ 2318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:10.294508 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:10.294529 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:10.295195 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:01:10.295198 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:10.295212 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:01:10.295250 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:10.295754 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:10.295760 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:10.295767 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 05:01:10.295922 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 05:01:10.295964 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2358,"thread_start_us":723,"threads_started":1}
I20260925 05:01:10.297950 4380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:10.289631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55156 (local address 127.30.247.1:41399)
0925 05:01:10.291969 (+ 2338us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:10.291978 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:10.292007 (+ 29us) server_negotiation.cc:407] Connection header received
0925 05:01:10.292076 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:10.292084 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:10.292348 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:10.293122 (+ 774us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:10.295141 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:10.295152 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:10.295167 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:01:10.295190 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:10.295244 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:01:10.295263 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:10.296674 (+ 1411us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:10.296746 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 05:01:10.296839 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2166,"thread_start_us":2041,"threads_started":1}
I20260925 05:01:10.301110 4379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:10.292155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:45055 (local address 127.0.0.1:46770)
0925 05:01:10.296375 (+ 4220us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:10.296392 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:10.296461 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:10.299468 (+ 3007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:10.299472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:10.299504 (+ 32us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:10.300052 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:01:10.300055 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:10.300070 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:01:10.300122 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:10.300725 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:10.300732 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:10.300737 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:01:10.300834 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 05:01:10.300881 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4117,"thread_start_us":6869,"threads_started":1}
I20260925 05:01:10.301384 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:10.292447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46770 (local address 127.30.247.3:45055)
0925 05:01:10.298979 (+ 6532us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:10.298989 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:10.299021 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:01:10.299086 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:10.299094 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:10.299280 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:10.299441 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:10.300482 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:10.300489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:10.300500 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:01:10.300515 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:10.300594 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:01:10.300615 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:10.300772 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:10.300847 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 05:01:10.300930 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6358,"thread_start_us":6197,"threads_started":1}
I20260925 05:01:10.302083 4171 tablet_service.cc:1467] Processing CreateTablet for tablet d8f7d1782b93407697ff7cd8fc37f4fb (DEFAULT_TABLE table=test-table [id=d07ee79674ba474c96e4b4a0160d317f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:01:10.302495 4040 tablet_service.cc:1467] Processing CreateTablet for tablet d8f7d1782b93407697ff7cd8fc37f4fb (DEFAULT_TABLE table=test-table [id=d07ee79674ba474c96e4b4a0160d317f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:01:10.303519 4171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f7d1782b93407697ff7cd8fc37f4fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:10.303622 4040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f7d1782b93407697ff7cd8fc37f4fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:10.304713 4301 tablet_service.cc:1467] Processing CreateTablet for tablet d8f7d1782b93407697ff7cd8fc37f4fb (DEFAULT_TABLE table=test-table [id=d07ee79674ba474c96e4b4a0160d317f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:01:10.305922 4301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f7d1782b93407697ff7cd8fc37f4fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:10.314345 4384 tablet_bootstrap.cc:492] T d8f7d1782b93407697ff7cd8fc37f4fb P 0453504661a240be9f932b749bdd7a12: Bootstrap starting.
I20260925 05:01:10.317517 4385 tablet_bootstrap.cc:492] T d8f7d1782b93407697ff7cd8fc37f4fb P 8f570660fe0f4a8c8e5d694c6bf7cefd: Bootstrap starting.
I20260925 05:01:10.317554 4384 tablet_bootstrap.cc:654] T d8f7d1782b93407697ff7cd8fc37f4fb P 0453504661a240be9f932b749bdd7a12: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:10.318727 4384 log.cc:824] T d8f7d1782b93407697ff7cd8fc37f4fb P 0453504661a240be9f932b749bdd7a12: Log is configured to *not* fsync() on all Append() calls
I20260925 05:01:10.320255 4386 tablet_bootstrap.cc:492] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823: Bootstrap starting.
I20260925 05:01:10.321199 4385 tablet_bootstrap.cc:654] T d8f7d1782b93407697ff7cd8fc37f4fb P 8f570660fe0f4a8c8e5d694c6bf7cefd: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:10.322093 4385 log.cc:824] T d8f7d1782b93407697ff7cd8fc37f4fb P 8f570660fe0f4a8c8e5d694c6bf7cefd: Log is configured to *not* fsync() on all Append() calls
I20260925 05:01:10.322711 4384 tablet_bootstrap.cc:492] T d8f7d1782b93407697ff7cd8fc37f4fb P 0453504661a240be9f932b749bdd7a12: No bootstrap required, opened a new log
I20260925 05:01:10.322983 4384 ts_tablet_manager.cc:1397] T d8f7d1782b93407697ff7cd8fc37f4fb P 0453504661a240be9f932b749bdd7a12: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 05:01:10.323608 4386 tablet_bootstrap.cc:654] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:10.324415 4385 tablet_bootstrap.cc:492] T d8f7d1782b93407697ff7cd8fc37f4fb P 8f570660fe0f4a8c8e5d694c6bf7cefd: No bootstrap required, opened a new log
I20260925 05:01:10.324764 4385 ts_tablet_manager.cc:1397] T d8f7d1782b93407697ff7cd8fc37f4fb P 8f570660fe0f4a8c8e5d694c6bf7cefd: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260925 05:01:10.324733 4386 log.cc:824] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823: Log is configured to *not* fsync() on all Append() calls
I20260925 05:01:10.327477 4386 tablet_bootstrap.cc:492] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823: No bootstrap required, opened a new log
I20260925 05:01:10.327766 4386 ts_tablet_manager.cc:1397] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260925 05:01:10.331984 4385 raft_consensus.cc:348] T d8f7d1782b93407697ff7cd8fc37f4fb P 8f570660fe0f4a8c8e5d694c6bf7cefd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0453504661a240be9f932b749bdd7a12" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 41399 } } peers { permanent_uuid: "8f570660fe0f4a8c8e5d694c6bf7cefd" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 45705 } } peers { permanent_uuid: "a6b5ee26c88d452880a02db12f312823" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45055 } }
I20260925 05:01:10.332324 4385 raft_consensus.cc:374] T d8f7d1782b93407697ff7cd8fc37f4fb P 8f570660fe0f4a8c8e5d694c6bf7cefd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:10.332417 4385 raft_consensus.cc:729] T d8f7d1782b93407697ff7cd8fc37f4fb P 8f570660fe0f4a8c8e5d694c6bf7cefd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f570660fe0f4a8c8e5d694c6bf7cefd, State: Initialized, Role: FOLLOWER
I20260925 05:01:10.333030 4385 consensus_queue.cc:260] T d8f7d1782b93407697ff7cd8fc37f4fb P 8f570660fe0f4a8c8e5d694c6bf7cefd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0453504661a240be9f932b749bdd7a12" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 41399 } } peers { permanent_uuid: "8f570660fe0f4a8c8e5d694c6bf7cefd" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 45705 } } peers { permanent_uuid: "a6b5ee26c88d452880a02db12f312823" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45055 } }
I20260925 05:01:10.333176 4384 raft_consensus.cc:348] T d8f7d1782b93407697ff7cd8fc37f4fb P 0453504661a240be9f932b749bdd7a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0453504661a240be9f932b749bdd7a12" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 41399 } } peers { permanent_uuid: "8f570660fe0f4a8c8e5d694c6bf7cefd" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 45705 } } peers { permanent_uuid: "a6b5ee26c88d452880a02db12f312823" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45055 } }
I20260925 05:01:10.333606 4384 raft_consensus.cc:374] T d8f7d1782b93407697ff7cd8fc37f4fb P 0453504661a240be9f932b749bdd7a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:10.333724 4384 raft_consensus.cc:729] T d8f7d1782b93407697ff7cd8fc37f4fb P 0453504661a240be9f932b749bdd7a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0453504661a240be9f932b749bdd7a12, State: Initialized, Role: FOLLOWER
I20260925 05:01:10.334095 4236 heartbeater.cc:503] Master 127.30.247.62:45863 was elected leader, sending a full tablet report...
I20260925 05:01:10.334383 4384 consensus_queue.cc:260] T d8f7d1782b93407697ff7cd8fc37f4fb P 0453504661a240be9f932b749bdd7a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0453504661a240be9f932b749bdd7a12" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 41399 } } peers { permanent_uuid: "8f570660fe0f4a8c8e5d694c6bf7cefd" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 45705 } } peers { permanent_uuid: "a6b5ee26c88d452880a02db12f312823" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45055 } }
I20260925 05:01:10.334681 4385 ts_tablet_manager.cc:1428] T d8f7d1782b93407697ff7cd8fc37f4fb P 8f570660fe0f4a8c8e5d694c6bf7cefd: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260925 05:01:10.335443 4105 heartbeater.cc:503] Master 127.30.247.62:45863 was elected leader, sending a full tablet report...
I20260925 05:01:10.335984 4384 ts_tablet_manager.cc:1428] T d8f7d1782b93407697ff7cd8fc37f4fb P 0453504661a240be9f932b749bdd7a12: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260925 05:01:10.337563 4386 raft_consensus.cc:348] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0453504661a240be9f932b749bdd7a12" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 41399 } } peers { permanent_uuid: "8f570660fe0f4a8c8e5d694c6bf7cefd" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 45705 } } peers { permanent_uuid: "a6b5ee26c88d452880a02db12f312823" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45055 } }
I20260925 05:01:10.337893 4386 raft_consensus.cc:374] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:10.338034 4386 raft_consensus.cc:729] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6b5ee26c88d452880a02db12f312823, State: Initialized, Role: FOLLOWER
I20260925 05:01:10.338696 4386 consensus_queue.cc:260] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0453504661a240be9f932b749bdd7a12" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 41399 } } peers { permanent_uuid: "8f570660fe0f4a8c8e5d694c6bf7cefd" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 45705 } } peers { permanent_uuid: "a6b5ee26c88d452880a02db12f312823" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45055 } }
I20260925 05:01:10.339745 4366 heartbeater.cc:503] Master 127.30.247.62:45863 was elected leader, sending a full tablet report...
I20260925 05:01:10.340457 4386 ts_tablet_manager.cc:1428] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
W20260925 05:01:10.399348 4106 tablet.cc:2369] T d8f7d1782b93407697ff7cd8fc37f4fb P 0453504661a240be9f932b749bdd7a12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:01:10.454129 4392 raft_consensus.cc:482] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:01:10.454486 4392 raft_consensus.cc:504] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0453504661a240be9f932b749bdd7a12" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 41399 } } peers { permanent_uuid: "8f570660fe0f4a8c8e5d694c6bf7cefd" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 45705 } } peers { permanent_uuid: "a6b5ee26c88d452880a02db12f312823" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45055 } }
I20260925 05:01:10.456394 4392 leader_election.cc:288] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0453504661a240be9f932b749bdd7a12 (127.30.247.1:41399), 8f570660fe0f4a8c8e5d694c6bf7cefd (127.30.247.2:45705)
I20260925 05:01:10.459141 4368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:10.456640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:41399 (local address 127.30.247.3:40823)
0925 05:01:10.456827 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:10.456847 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:10.456956 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:10.457574 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:10.457580 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:10.457604 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:10.458239 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:01:10.458244 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:10.458264 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:01:10.458317 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:10.458716 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:10.458722 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:10.458728 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 05:01:10.458845 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 05:01:10.458887 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 05:01:10.459203 4380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:10.457113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:40823 (local address 127.30.247.1:41399)
0925 05:01:10.457295 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:10.457300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:10.457315 (+ 15us) server_negotiation.cc:407] Connection header received
0925 05:01:10.457366 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:10.457370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:10.457413 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:10.457496 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:10.458481 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:10.458489 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:10.458494 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:01:10.458516 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:10.458557 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:01:10.458574 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:10.458728 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:10.458852 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 05:01:10.458910 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 05:01:10.459455 4381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:10.456977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:44833 (local address 127.30.247.2:45705)
0925 05:01:10.457148 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:10.457155 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:10.457647 (+ 492us) server_negotiation.cc:407] Connection header received
0925 05:01:10.457743 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:10.457746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:10.457794 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:10.457916 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:10.458845 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:10.458850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:10.458854 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:01:10.458873 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:10.458908 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:01:10.458922 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:10.459043 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:10.459133 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 05:01:10.459190 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 05:01:10.459641 4393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:10.456838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:45705 (local address 127.30.247.3:44833)
0925 05:01:10.457514 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:10.457535 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:10.457629 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:10.457940 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:10.457943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:10.457959 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:10.458603 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:01:10.458605 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:10.458616 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:01:10.458647 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:10.459032 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:10.459036 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:10.459040 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:01:10.459158 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 05:01:10.459218 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":412,"threads_started":1}
I20260925 05:01:10.460311 4060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f7d1782b93407697ff7cd8fc37f4fb" candidate_uuid: "a6b5ee26c88d452880a02db12f312823" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0453504661a240be9f932b749bdd7a12" is_pre_election: true
I20260925 05:01:10.460611 4191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f7d1782b93407697ff7cd8fc37f4fb" candidate_uuid: "a6b5ee26c88d452880a02db12f312823" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f570660fe0f4a8c8e5d694c6bf7cefd" is_pre_election: true
I20260925 05:01:10.460834 4060 raft_consensus.cc:2454] T d8f7d1782b93407697ff7cd8fc37f4fb P 0453504661a240be9f932b749bdd7a12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6b5ee26c88d452880a02db12f312823 in term 0.
I20260925 05:01:10.461091 4191 raft_consensus.cc:2454] T d8f7d1782b93407697ff7cd8fc37f4fb P 8f570660fe0f4a8c8e5d694c6bf7cefd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6b5ee26c88d452880a02db12f312823 in term 0.
I20260925 05:01:10.461565 4257 leader_election.cc:302] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0453504661a240be9f932b749bdd7a12, a6b5ee26c88d452880a02db12f312823; no voters:
I20260925 05:01:10.461987 4392 raft_consensus.cc:2781] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:01:10.462134 4392 raft_consensus.cc:482] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:01:10.462234 4392 raft_consensus.cc:3037] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:10.464740 4392 raft_consensus.cc:504] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0453504661a240be9f932b749bdd7a12" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 41399 } } peers { permanent_uuid: "8f570660fe0f4a8c8e5d694c6bf7cefd" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 45705 } } peers { permanent_uuid: "a6b5ee26c88d452880a02db12f312823" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45055 } }
I20260925 05:01:10.465242 4392 leader_election.cc:288] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823 [CANDIDATE]: Term 1 election: Requested vote from peers 0453504661a240be9f932b749bdd7a12 (127.30.247.1:41399), 8f570660fe0f4a8c8e5d694c6bf7cefd (127.30.247.2:45705)
I20260925 05:01:10.465699 4191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f7d1782b93407697ff7cd8fc37f4fb" candidate_uuid: "a6b5ee26c88d452880a02db12f312823" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f570660fe0f4a8c8e5d694c6bf7cefd"
I20260925 05:01:10.465699 4060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f7d1782b93407697ff7cd8fc37f4fb" candidate_uuid: "a6b5ee26c88d452880a02db12f312823" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0453504661a240be9f932b749bdd7a12"
I20260925 05:01:10.465894 4191 raft_consensus.cc:3037] T d8f7d1782b93407697ff7cd8fc37f4fb P 8f570660fe0f4a8c8e5d694c6bf7cefd [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:10.465911 4060 raft_consensus.cc:3037] T d8f7d1782b93407697ff7cd8fc37f4fb P 0453504661a240be9f932b749bdd7a12 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:10.469089 4060 raft_consensus.cc:2454] T d8f7d1782b93407697ff7cd8fc37f4fb P 0453504661a240be9f932b749bdd7a12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6b5ee26c88d452880a02db12f312823 in term 1.
I20260925 05:01:10.469085 4191 raft_consensus.cc:2454] T d8f7d1782b93407697ff7cd8fc37f4fb P 8f570660fe0f4a8c8e5d694c6bf7cefd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6b5ee26c88d452880a02db12f312823 in term 1.
I20260925 05:01:10.469482 4255 leader_election.cc:302] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f570660fe0f4a8c8e5d694c6bf7cefd, a6b5ee26c88d452880a02db12f312823; no voters:
I20260925 05:01:10.469835 4392 raft_consensus.cc:2781] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:01:10.470219 4392 raft_consensus.cc:686] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823 [term 1 LEADER]: Becoming Leader. State: Replica: a6b5ee26c88d452880a02db12f312823, State: Running, Role: LEADER
I20260925 05:01:10.470733 4392 consensus_queue.cc:237] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0453504661a240be9f932b749bdd7a12" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 41399 } } peers { permanent_uuid: "8f570660fe0f4a8c8e5d694c6bf7cefd" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 45705 } } peers { permanent_uuid: "a6b5ee26c88d452880a02db12f312823" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45055 } }
I20260925 05:01:10.475045 3918 catalog_manager.cc:5495] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6b5ee26c88d452880a02db12f312823 (127.30.247.3). New cstate: current_term: 1 leader_uuid: "a6b5ee26c88d452880a02db12f312823" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0453504661a240be9f932b749bdd7a12" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 41399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f570660fe0f4a8c8e5d694c6bf7cefd" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 45705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6b5ee26c88d452880a02db12f312823" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45055 } health_report { overall_health: HEALTHY } } }
W20260925 05:01:10.489097 4367 tablet.cc:2369] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:01:10.532008 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:10.530188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46786 (local address 127.30.247.3:45055)
0925 05:01:10.530377 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:10.530381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:10.530398 (+ 17us) server_negotiation.cc:407] Connection header received
0925 05:01:10.530484 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:10.530488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:10.530561 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:10.530649 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:10.531514 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:10.531518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:10.531521 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:01:10.531536 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:10.531567 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:01:10.531578 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:10.531674 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:10.531784 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 05:01:10.531829 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 05:01:10.539813 4191 raft_consensus.cc:1261] T d8f7d1782b93407697ff7cd8fc37f4fb P 8f570660fe0f4a8c8e5d694c6bf7cefd [term 1 FOLLOWER]: Refusing update from remote peer a6b5ee26c88d452880a02db12f312823: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:01:10.539827 4060 raft_consensus.cc:1261] T d8f7d1782b93407697ff7cd8fc37f4fb P 0453504661a240be9f932b749bdd7a12 [term 1 FOLLOWER]: Refusing update from remote peer a6b5ee26c88d452880a02db12f312823: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:01:10.540763 4394 consensus_queue.cc:1035] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0453504661a240be9f932b749bdd7a12" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 41399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:01:10.541076 4392 consensus_queue.cc:1035] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f570660fe0f4a8c8e5d694c6bf7cefd" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 45705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:01:10.566288 3918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:01:10.566574 3918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:01:10.570183 3918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d9f20712d9824b3f8618b8dd75d81b9e: Sending DeleteTablet for 3 replicas of tablet d8f7d1782b93407697ff7cd8fc37f4fb
I20260925 05:01:10.571276 4171 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f7d1782b93407697ff7cd8fc37f4fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:01:10 UTC) from {username='slave'} at 127.0.0.1:57610
I20260925 05:01:10.571389 4040 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f7d1782b93407697ff7cd8fc37f4fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:01:10 UTC) from {username='slave'} at 127.0.0.1:55156
I20260925 05:01:10.571533 4301 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f7d1782b93407697ff7cd8fc37f4fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:01:10 UTC) from {username='slave'} at 127.0.0.1:46770
I20260925 05:01:10.573769 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 3978
I20260925 05:01:10.601558 4408 tablet_replica.cc:330] T d8f7d1782b93407697ff7cd8fc37f4fb P 0453504661a240be9f932b749bdd7a12: stopping tablet replica
I20260925 05:01:10.603152 4380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:10.575408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55160 (local address 127.30.247.1:41399)
0925 05:01:10.575628 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:10.575637 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:10.599579 (+ 23942us) server_negotiation.cc:407] Connection header received
0925 05:01:10.600116 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:10.600124 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:10.600263 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:10.600353 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:10.602176 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:10.602185 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:10.602192 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:01:10.602217 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:10.602263 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:01:10.602284 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:10.602400 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:10.602662 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 05:01:10.602738 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 05:01:10.604384 4100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:10.616375 4407 tablet_replica.cc:330] T d8f7d1782b93407697ff7cd8fc37f4fb P 8f570660fe0f4a8c8e5d694c6bf7cefd: stopping tablet replica
I20260925 05:01:10.616874 4407 raft_consensus.cc:2229] T d8f7d1782b93407697ff7cd8fc37f4fb P 8f570660fe0f4a8c8e5d694c6bf7cefd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:01:10.617611 4407 raft_consensus.cc:2258] T d8f7d1782b93407697ff7cd8fc37f4fb P 8f570660fe0f4a8c8e5d694c6bf7cefd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:01:10.619994 4409 tablet_replica.cc:330] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823: stopping tablet replica
I20260925 05:01:10.620007 4407 ts_tablet_manager.cc:1905] T d8f7d1782b93407697ff7cd8fc37f4fb P 8f570660fe0f4a8c8e5d694c6bf7cefd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:01:10.620380 4409 raft_consensus.cc:2229] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:01:10.622289 4409 raft_consensus.cc:2258] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:01:10.624496 4409 ts_tablet_manager.cc:1905] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:01:10.628186 4407 ts_tablet_manager.cc:1918] T d8f7d1782b93407697ff7cd8fc37f4fb P 8f570660fe0f4a8c8e5d694c6bf7cefd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:01:10.628384 4407 log.cc:1196] T d8f7d1782b93407697ff7cd8fc37f4fb P 8f570660fe0f4a8c8e5d694c6bf7cefd: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-1/wal/wals/d8f7d1782b93407697ff7cd8fc37f4fb
I20260925 05:01:10.629025 4407 ts_tablet_manager.cc:1939] T d8f7d1782b93407697ff7cd8fc37f4fb P 8f570660fe0f4a8c8e5d694c6bf7cefd: Deleting consensus metadata
I20260925 05:01:10.630847 3903 catalog_manager.cc:4841] TS 8f570660fe0f4a8c8e5d694c6bf7cefd (127.30.247.2:45705): tablet d8f7d1782b93407697ff7cd8fc37f4fb (table test-table [id=d07ee79674ba474c96e4b4a0160d317f]) successfully deleted
I20260925 05:01:10.631541 4409 ts_tablet_manager.cc:1918] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:01:10.631712 4409 log.cc:1196] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312430481417-31708-0/minicluster-data/ts-2/wal/wals/d8f7d1782b93407697ff7cd8fc37f4fb
I20260925 05:01:10.632159 4409 ts_tablet_manager.cc:1939] T d8f7d1782b93407697ff7cd8fc37f4fb P a6b5ee26c88d452880a02db12f312823: Deleting consensus metadata
I20260925 05:01:10.633522 3905 catalog_manager.cc:4841] TS a6b5ee26c88d452880a02db12f312823 (127.30.247.3:45055): tablet d8f7d1782b93407697ff7cd8fc37f4fb (table test-table [id=d07ee79674ba474c96e4b4a0160d317f]) successfully deleted
I20260925 05:01:10.614410 4408 raft_consensus.cc:2229] T d8f7d1782b93407697ff7cd8fc37f4fb P 0453504661a240be9f932b749bdd7a12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:01:10.753742 4408 raft_consensus.cc:2258] T d8f7d1782b93407697ff7cd8fc37f4fb P 0453504661a240be9f932b749bdd7a12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:01:10.754549 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 3978
I20260925 05:01:10.771664 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 4109
W20260925 05:01:10.771649 3905 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.247.1:41399 (error 108)
I20260925 05:01:10.774070 4379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:10.773657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:41399 (local address 127.0.0.1:55164)
0925 05:01:10.773821 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:10.773919 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.1:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 05:01:10.774438 3905 catalog_manager.cc:4584] TS 0453504661a240be9f932b749bdd7a12 (127.30.247.1:41399): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d8f7d1782b93407697ff7cd8fc37f4fb: Network error: Client connection negotiation failed: client connection to 127.30.247.1:41399: connect: Connection refused (error 111)
I20260925 05:01:10.775205 4381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:10.772472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.30.247.2:45705)
0925 05:01:10.772689 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:10.772696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:10.772718 (+ 22us) server_negotiation.cc:407] Connection header received
0925 05:01:10.772912 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:10.772918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:10.773025 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:10.773135 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:10.774523 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:10.774529 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:10.774538 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:01:10.774553 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:10.774606 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:01:10.774625 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:10.774743 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:10.774894 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 05:01:10.774955 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 05:01:10.775862 4231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:10.828034 4379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:10.827515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:41399 (local address 127.0.0.1:55170)
0925 05:01:10.827746 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:10.827867 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.1:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 05:01:10.829370 4379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:10.828928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:41399 (local address 127.0.0.1:55180)
0925 05:01:10.829108 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:10.829206 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.1:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 05:01:10.865633 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 4109
I20260925 05:01:10.884092 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 4239
I20260925 05:01:10.887231 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:10.884905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46800 (local address 127.30.247.3:45055)
0925 05:01:10.885069 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:10.885075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:10.885167 (+ 92us) server_negotiation.cc:407] Connection header received
0925 05:01:10.885360 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:10.885366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:10.885432 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:10.885542 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:10.886494 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:10.886500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:10.886507 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:01:10.886533 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:10.886579 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:01:10.886595 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:10.886740 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:10.886850 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 05:01:10.886909 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 05:01:10.888168 4361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:10.890091 4379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:10.889471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:41399 (local address 127.0.0.1:55192)
0925 05:01:10.889741 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:10.889874 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.1:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 05:01:10.893415 4379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:10.893073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:41399 (local address 127.0.0.1:55202)
0925 05:01:10.893213 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:10.893294 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.1:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 05:01:10.974170 4379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:10.973656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:41399 (local address 127.0.0.1:55204)
0925 05:01:10.973911 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:10.974016 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.1:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260925 05:01:10.975412 4379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:10.974980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:41399 (local address 127.0.0.1:55208)
0925 05:01:10.975194 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:10.975269 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.1:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260925 05:01:11.008370 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 4239
I20260925 05:01:11.027788 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 3887
I20260925 05:01:11.028955 3948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:11.131438 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 3887
2026-09-25T05:01:11Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2703 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 05:01:11.173480 31708 mini_oidc.cc:152] Starting JWKS server
I20260925 05:01:11.176065 31708 webserver.cc:460] Webserver started at https://127.0.0.1:33641/ using document root <none> and password file <none>
I20260925 05:01:11.176391 31708 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 05:01:11.177088 31708 webserver.cc:460] Webserver started at http://127.0.0.1:44071/ using document root <none> and password file <none>
2026-09-25T05:01:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:01:11Z Disabled control of system clock
I20260925 05:01:11.202483 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:45503
--webserver_interface=127.30.247.62
--webserver_port=0
--builtin_ntp_servers=127.30.247.20:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:45503
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33641/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312430481417-31708-0/testchainca.cert with env {}
W20260925 05:01:11.517150 4421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:11.517431 4421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:11.517517 4421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:11.524474 4421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:01:11.524628 4421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:11.524690 4421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:11.524752 4421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:01:11.524799 4421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 05:01:11.524853 4421 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 05:01:11.524892 4421 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33641/jwks.json
I20260925 05:01:11.534544 4421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:45503
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:45503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33641/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312430481417-31708-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:11.535692 4421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:11.536911 4421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:11.548269 4426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:11.548256 4427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:11.548445 4429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:11.549440 4421 server_base.cc:1034] running on GCE node
I20260925 05:01:11.550112 4421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:11.551362 4421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:11.552606 4421 hybrid_clock.cc:648] HybridClock initialized: now 1790312471552509 us; error 47 us; skew 500 ppm
I20260925 05:01:11.552915 4421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:01:11.555091 4421 webserver.cc:460] Webserver started at http://127.30.247.62:35089/ using document root <none> and password file <none>
I20260925 05:01:11.555743 4421 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:11.555832 4421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:11.556059 4421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:11.558825 4421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "71547838ba0d486ea588e90a93fbf09f"
format_stamp: "Formatted at 2026-09-25 05:01:11 on dist-test-slave-5n86"
server_key: "fd6168db3c3c028da42e35958a2b63b1"
server_key_iv: "45dfcc808fd42f83cb68d11670e11582"
server_key_version: "encryptionkey@0"
I20260925 05:01:11.559526 4421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "71547838ba0d486ea588e90a93fbf09f"
format_stamp: "Formatted at 2026-09-25 05:01:11 on dist-test-slave-5n86"
server_key: "fd6168db3c3c028da42e35958a2b63b1"
server_key_iv: "45dfcc808fd42f83cb68d11670e11582"
server_key_version: "encryptionkey@0"
I20260925 05:01:11.564193 4421 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260925 05:01:11.567585 4435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:11.569052 4421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:01:11.569234 4421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "71547838ba0d486ea588e90a93fbf09f"
format_stamp: "Formatted at 2026-09-25 05:01:11 on dist-test-slave-5n86"
server_key: "fd6168db3c3c028da42e35958a2b63b1"
server_key_iv: "45dfcc808fd42f83cb68d11670e11582"
server_key_version: "encryptionkey@0"
I20260925 05:01:11.569427 4421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:11.593169 4421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:11.593953 4421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:11.594269 4421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:11.611980 4421 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:45503
I20260925 05:01:11.612000 4486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:45503 every 8 connection(s)
I20260925 05:01:11.613219 4421 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:01:11.616667 4487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:11.618919 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 4421
I20260925 05:01:11.619122 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:01:11.619891 31708 external_mini_cluster.cc:1428] Setting key d74b42f1161628a78e041fbfa001499b
I20260925 05:01:11.631282 4487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71547838ba0d486ea588e90a93fbf09f: Bootstrap starting.
I20260925 05:01:11.635087 4487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71547838ba0d486ea588e90a93fbf09f: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:11.636343 4487 log.cc:824] T 00000000000000000000000000000000 P 71547838ba0d486ea588e90a93fbf09f: Log is configured to *not* fsync() on all Append() calls
I20260925 05:01:11.638763 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:11.622745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57642 (local address 127.30.247.62:45503)
0925 05:01:11.623963 (+ 1218us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:11.623986 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:11.624033 (+ 47us) server_negotiation.cc:407] Connection header received
0925 05:01:11.626374 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:11.626411 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:11.627013 (+ 602us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:11.627450 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:11.628621 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:11.631042 (+ 2421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:11.632955 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:11.633947 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:11.634119 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:11.635932 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:11.635943 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:11.635963 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:01:11.636690 (+ 727us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:11.636813 (+ 123us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:01:11.636844 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:11.636980 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:11.637265 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 05:01:11.637599 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":400,"threads_started":1}
I20260925 05:01:11.639556 4487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71547838ba0d486ea588e90a93fbf09f: No bootstrap required, opened a new log
I20260925 05:01:11.643213 31708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:01:11.650182 4487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71547838ba0d486ea588e90a93fbf09f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71547838ba0d486ea588e90a93fbf09f" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45503 } }
I20260925 05:01:11.650674 4487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71547838ba0d486ea588e90a93fbf09f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:11.650768 4487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71547838ba0d486ea588e90a93fbf09f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71547838ba0d486ea588e90a93fbf09f, State: Initialized, Role: FOLLOWER
I20260925 05:01:11.651398 4487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71547838ba0d486ea588e90a93fbf09f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71547838ba0d486ea588e90a93fbf09f" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45503 } }
I20260925 05:01:11.651654 4487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71547838ba0d486ea588e90a93fbf09f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:01:11.651782 4487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71547838ba0d486ea588e90a93fbf09f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:01:11.651937 4487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71547838ba0d486ea588e90a93fbf09f [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:11.655053 4487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71547838ba0d486ea588e90a93fbf09f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71547838ba0d486ea588e90a93fbf09f" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45503 } }
I20260925 05:01:11.655553 4487 leader_election.cc:302] T 00000000000000000000000000000000 P 71547838ba0d486ea588e90a93fbf09f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71547838ba0d486ea588e90a93fbf09f; no voters:
I20260925 05:01:11.656132 4487 leader_election.cc:288] T 00000000000000000000000000000000 P 71547838ba0d486ea588e90a93fbf09f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:01:11.656332 4492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71547838ba0d486ea588e90a93fbf09f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:01:11.657169 4492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71547838ba0d486ea588e90a93fbf09f [term 1 LEADER]: Becoming Leader. State: Replica: 71547838ba0d486ea588e90a93fbf09f, State: Running, Role: LEADER
I20260925 05:01:11.657735 4492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71547838ba0d486ea588e90a93fbf09f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71547838ba0d486ea588e90a93fbf09f" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45503 } }
I20260925 05:01:11.658510 4487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71547838ba0d486ea588e90a93fbf09f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:01:11.660599 4493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71547838ba0d486ea588e90a93fbf09f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71547838ba0d486ea588e90a93fbf09f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71547838ba0d486ea588e90a93fbf09f" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45503 } } }
I20260925 05:01:11.661021 4493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71547838ba0d486ea588e90a93fbf09f [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:11.661439 4494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71547838ba0d486ea588e90a93fbf09f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71547838ba0d486ea588e90a93fbf09f. Latest consensus state: current_term: 1 leader_uuid: "71547838ba0d486ea588e90a93fbf09f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71547838ba0d486ea588e90a93fbf09f" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45503 } } }
I20260925 05:01:11.661779 4494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71547838ba0d486ea588e90a93fbf09f [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:11.664067 4500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:01:11.669032 4500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:01:11.678223 4500 catalog_manager.cc:1347] Generated new cluster ID: 0899c759881545909aace6099ceab3bb
I20260925 05:01:11.678326 4500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:01:11.691473 4500 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:01:11.693236 4500 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:01:11.713713 4500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71547838ba0d486ea588e90a93fbf09f: Generated new TSK 0
I20260925 05:01:11.714510 4500 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 05:01:11.756764 4517 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 05:01:11.756954 4517 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 05:01:11.757966 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:11.735894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57656 (local address 127.30.247.62:45503)
0925 05:01:11.736105 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:11.736111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:11.736715 (+ 604us) server_negotiation.cc:407] Connection header received
0925 05:01:11.737020 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:11.737025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:11.737111 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:11.737266 (+ 155us) server_negotiation.cc:226] Negotiated authn=JWT
0925 05:01:11.737821 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:11.739018 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:11.740694 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:11.741376 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:11.741509 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:11.742031 (+ 522us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 05:01:11.757302 (+ 15271us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33641/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 05:01:11.757671 (+ 369us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57656: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33641/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":72}
W20260925 05:01:11.758082 4490 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57656: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33641/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 05:01:11.759420 4516 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:11.735793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:45503 (local address 127.0.0.1:57656)
0925 05:01:11.736611 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:11.736628 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:11.736782 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:11.737311 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:11.737315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:11.737325 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0925 05:01:11.737660 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:11.737668 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:11.739165 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:11.739170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:11.740495 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:11.740506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:11.741564 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:11.741568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:11.741702 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:11.741758 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 05:01:11.757609 (+ 15851us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33641/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 05:01:11.757899 (+ 290us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.247.62:45503: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33641/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":396,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.30.247.62:45503: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33641/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 05:01:11.761085 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 4421
I20260925 05:01:11.762290 4482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:11.857599 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 4421
2026-09-25T05:01:11Z chronyd exiting
I20260925 05:01:11.908010 31708 test_util.cc:175] -----------------------------------------------
I20260925 05:01:11.908186 31708 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312430481417-31708-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (738 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 05:01:11.911849 31708 mini_oidc.cc:152] Starting JWKS server
I20260925 05:01:11.913681 31708 webserver.cc:460] Webserver started at https://127.0.0.1:42849/ using document root <none> and password file <none>
I20260925 05:01:11.913918 31708 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 05:01:11.914580 31708 webserver.cc:460] Webserver started at http://127.0.0.1:46697/ using document root <none> and password file <none>
2026-09-25T05:01:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:01:11Z Disabled control of system clock
I20260925 05:01:11.939037 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:36329
--webserver_interface=127.30.247.62
--webserver_port=0
--builtin_ntp_servers=127.30.247.20:44723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:36329
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42849/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312430481417-31708-0/testchainca.cert with env {}
W20260925 05:01:12.249557 4524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:12.249822 4524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:12.249905 4524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:12.256893 4524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:01:12.257006 4524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:12.257045 4524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:12.257117 4524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:01:12.257161 4524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 05:01:12.257215 4524 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 05:01:12.257253 4524 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42849/jwks.json
I20260925 05:01:12.267066 4524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:44723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:36329
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:36329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42849/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312430481417-31708-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:12.268431 4524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:12.269758 4524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:12.281494 4529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:12.281461 4530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:12.281422 4532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:12.282581 4524 server_base.cc:1034] running on GCE node
I20260925 05:01:12.283335 4524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:12.284750 4524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:12.286010 4524 hybrid_clock.cc:648] HybridClock initialized: now 1790312472285966 us; error 54 us; skew 500 ppm
I20260925 05:01:12.286403 4524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:01:12.297216 4524 webserver.cc:460] Webserver started at http://127.30.247.62:46307/ using document root <none> and password file <none>
I20260925 05:01:12.297936 4524 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:12.298084 4524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:12.298427 4524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:12.301343 4524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "27be3f09fcdd41a3a969cc3b6049e35d"
format_stamp: "Formatted at 2026-09-25 05:01:12 on dist-test-slave-5n86"
server_key: "1beaf9a0ab22c9e7d9fed7516c4eb0c0"
server_key_iv: "1709455065416feed91db5988aa2e465"
server_key_version: "encryptionkey@0"
I20260925 05:01:12.302115 4524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "27be3f09fcdd41a3a969cc3b6049e35d"
format_stamp: "Formatted at 2026-09-25 05:01:12 on dist-test-slave-5n86"
server_key: "1beaf9a0ab22c9e7d9fed7516c4eb0c0"
server_key_iv: "1709455065416feed91db5988aa2e465"
server_key_version: "encryptionkey@0"
I20260925 05:01:12.307147 4524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:01:12.310678 4538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:12.312043 4524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260925 05:01:12.312260 4524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "27be3f09fcdd41a3a969cc3b6049e35d"
format_stamp: "Formatted at 2026-09-25 05:01:12 on dist-test-slave-5n86"
server_key: "1beaf9a0ab22c9e7d9fed7516c4eb0c0"
server_key_iv: "1709455065416feed91db5988aa2e465"
server_key_version: "encryptionkey@0"
I20260925 05:01:12.312500 4524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:12.344928 4524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:12.345762 4524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:12.346091 4524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:12.364091 4524 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:36329
I20260925 05:01:12.364178 4589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:36329 every 8 connection(s)
I20260925 05:01:12.365401 4524 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:01:12.369086 4590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:12.374552 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 4524
I20260925 05:01:12.374737 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:01:12.375401 31708 external_mini_cluster.cc:1428] Setting key 31c0d38a8108e3cdf3d4fd7b46649aea
I20260925 05:01:12.379868 4590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27be3f09fcdd41a3a969cc3b6049e35d: Bootstrap starting.
I20260925 05:01:12.383666 4590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27be3f09fcdd41a3a969cc3b6049e35d: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:12.384796 4590 log.cc:824] T 00000000000000000000000000000000 P 27be3f09fcdd41a3a969cc3b6049e35d: Log is configured to *not* fsync() on all Append() calls
I20260925 05:01:12.387879 4590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27be3f09fcdd41a3a969cc3b6049e35d: No bootstrap required, opened a new log
I20260925 05:01:12.390728 4593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:12.377354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46710 (local address 127.30.247.62:36329)
0925 05:01:12.378325 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:12.378339 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:12.378387 (+ 48us) server_negotiation.cc:407] Connection header received
0925 05:01:12.380643 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:12.380668 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:12.381231 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:12.381692 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:12.382204 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:12.384141 (+ 1937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:12.385782 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:12.386667 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:12.386831 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:12.387974 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:12.387985 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:12.388002 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:01:12.388744 (+ 742us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:12.388838 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:01:12.388862 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:12.388975 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:12.389206 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 05:01:12.389517 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":289,"threads_started":1}
I20260925 05:01:12.394515 31708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:01:12.396662 4590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27be3f09fcdd41a3a969cc3b6049e35d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27be3f09fcdd41a3a969cc3b6049e35d" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36329 } }
I20260925 05:01:12.397059 4590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27be3f09fcdd41a3a969cc3b6049e35d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:12.397177 4590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27be3f09fcdd41a3a969cc3b6049e35d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27be3f09fcdd41a3a969cc3b6049e35d, State: Initialized, Role: FOLLOWER
I20260925 05:01:12.397748 4590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27be3f09fcdd41a3a969cc3b6049e35d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27be3f09fcdd41a3a969cc3b6049e35d" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36329 } }
I20260925 05:01:12.397989 4590 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27be3f09fcdd41a3a969cc3b6049e35d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:01:12.398133 4590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27be3f09fcdd41a3a969cc3b6049e35d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:01:12.398311 4590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27be3f09fcdd41a3a969cc3b6049e35d [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:12.400305 4590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27be3f09fcdd41a3a969cc3b6049e35d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27be3f09fcdd41a3a969cc3b6049e35d" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36329 } }
I20260925 05:01:12.400835 4590 leader_election.cc:302] T 00000000000000000000000000000000 P 27be3f09fcdd41a3a969cc3b6049e35d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27be3f09fcdd41a3a969cc3b6049e35d; no voters:
I20260925 05:01:12.401376 4590 leader_election.cc:288] T 00000000000000000000000000000000 P 27be3f09fcdd41a3a969cc3b6049e35d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:01:12.401516 4595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27be3f09fcdd41a3a969cc3b6049e35d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:01:12.402292 4595 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27be3f09fcdd41a3a969cc3b6049e35d [term 1 LEADER]: Becoming Leader. State: Replica: 27be3f09fcdd41a3a969cc3b6049e35d, State: Running, Role: LEADER
I20260925 05:01:12.402822 4595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27be3f09fcdd41a3a969cc3b6049e35d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27be3f09fcdd41a3a969cc3b6049e35d" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36329 } }
I20260925 05:01:12.403486 4590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27be3f09fcdd41a3a969cc3b6049e35d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:01:12.405596 4596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27be3f09fcdd41a3a969cc3b6049e35d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27be3f09fcdd41a3a969cc3b6049e35d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27be3f09fcdd41a3a969cc3b6049e35d" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36329 } } }
I20260925 05:01:12.405956 4596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27be3f09fcdd41a3a969cc3b6049e35d [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:12.406471 4597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27be3f09fcdd41a3a969cc3b6049e35d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27be3f09fcdd41a3a969cc3b6049e35d. Latest consensus state: current_term: 1 leader_uuid: "27be3f09fcdd41a3a969cc3b6049e35d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27be3f09fcdd41a3a969cc3b6049e35d" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36329 } } }
I20260925 05:01:12.406838 4597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27be3f09fcdd41a3a969cc3b6049e35d [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:12.407021 4601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:01:12.417761 4601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:01:12.427608 4601 catalog_manager.cc:1347] Generated new cluster ID: 4777e232f65f41a98992e68003a9d39c
I20260925 05:01:12.427706 4601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:01:12.465644 4601 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:01:12.467480 4601 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:01:12.477591 4601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27be3f09fcdd41a3a969cc3b6049e35d: Generated new TSK 0
I20260925 05:01:12.478561 4601 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 05:01:12.516254 4621 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 05:01:12.516381 4621 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 05:01:12.517608 4593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:12.497679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46714 (local address 127.30.247.62:36329)
0925 05:01:12.497866 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:12.497873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:12.498294 (+ 421us) server_negotiation.cc:407] Connection header received
0925 05:01:12.498484 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:12.498489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:12.498583 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:12.498730 (+ 147us) server_negotiation.cc:226] Negotiated authn=JWT
0925 05:01:12.499163 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:12.500258 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:12.501733 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:12.502437 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:12.502567 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:12.503018 (+ 451us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 05:01:12.516932 (+ 13914us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 05:01:12.517326 (+ 394us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46714: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 05:01:12.517719 4593 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46714: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 05:01:12.517717 4620 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:12.497546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36329 (local address 127.0.0.1:46714)
0925 05:01:12.498191 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:12.498210 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:12.498325 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:12.498747 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:12.498751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:12.498760 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 05:01:12.499009 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:12.499016 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:12.500426 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:12.500430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:12.501594 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:12.501606 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:12.502607 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:12.502611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:12.502740 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:12.502772 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 05:01:12.517219 (+ 14447us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 05:01:12.517408 (+ 189us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.247.62:36329: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":372,"threads_started":1}
I20260925 05:01:12.518988 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 4524
I20260925 05:01:12.520076 4585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:12.614081 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 4524
2026-09-25T05:01:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (757 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 05:01:12.669422 31708 mini_oidc.cc:152] Starting JWKS server
I20260925 05:01:12.671248 31708 webserver.cc:460] Webserver started at https://127.0.0.1:44405/ using document root <none> and password file <none>
I20260925 05:01:12.671504 31708 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 05:01:12.672111 31708 webserver.cc:460] Webserver started at http://127.0.0.1:43377/ using document root <none> and password file <none>
2026-09-25T05:01:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:01:12Z Disabled control of system clock
I20260925 05:01:12.698079 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:34079
--webserver_interface=127.30.247.62
--webserver_port=0
--builtin_ntp_servers=127.30.247.20:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:34079
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44405/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 05:01:13.007750 4628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:13.008013 4628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:13.008175 4628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:13.015172 4628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:01:13.015270 4628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:13.015349 4628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:13.015442 4628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:01:13.015532 4628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 05:01:13.015643 4628 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 05:01:13.015722 4628 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44405/jwks.json
I20260925 05:01:13.024484 4628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:34079
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:34079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44405/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:13.025839 4628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:13.027616 4628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:13.039422 4634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:13.039402 4633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:13.039613 4636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:13.040235 4628 server_base.cc:1034] running on GCE node
I20260925 05:01:13.041002 4628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:13.042302 4628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:13.043524 4628 hybrid_clock.cc:648] HybridClock initialized: now 1790312473043467 us; error 48 us; skew 500 ppm
I20260925 05:01:13.043905 4628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:01:13.046095 4628 webserver.cc:460] Webserver started at http://127.30.247.62:42281/ using document root <none> and password file <none>
I20260925 05:01:13.046785 4628 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:13.046906 4628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:13.047233 4628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:13.050046 4628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "559866f5f3594469b2afa609295ec271"
format_stamp: "Formatted at 2026-09-25 05:01:13 on dist-test-slave-5n86"
server_key: "a17872b7542c4e54c639cae53a9f7540"
server_key_iv: "130c76b97d94084b299d265982fed756"
server_key_version: "encryptionkey@0"
I20260925 05:01:13.050836 4628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "559866f5f3594469b2afa609295ec271"
format_stamp: "Formatted at 2026-09-25 05:01:13 on dist-test-slave-5n86"
server_key: "a17872b7542c4e54c639cae53a9f7540"
server_key_iv: "130c76b97d94084b299d265982fed756"
server_key_version: "encryptionkey@0"
I20260925 05:01:13.055721 4628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260925 05:01:13.059090 4642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:13.060416 4628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:01:13.060621 4628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "559866f5f3594469b2afa609295ec271"
format_stamp: "Formatted at 2026-09-25 05:01:13 on dist-test-slave-5n86"
server_key: "a17872b7542c4e54c639cae53a9f7540"
server_key_iv: "130c76b97d94084b299d265982fed756"
server_key_version: "encryptionkey@0"
I20260925 05:01:13.060845 4628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:13.097659 4628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:13.098518 4628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:13.098839 4628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:13.116963 4628 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:34079
I20260925 05:01:13.117009 4693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:34079 every 8 connection(s)
I20260925 05:01:13.118202 4628 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:01:13.121760 4694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:13.124315 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 4628
I20260925 05:01:13.124595 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:01:13.125198 31708 external_mini_cluster.cc:1428] Setting key 8b52589d7e06647eec13e0cf10b55f6a
I20260925 05:01:13.132928 4694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 559866f5f3594469b2afa609295ec271: Bootstrap starting.
I20260925 05:01:13.135918 4694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 559866f5f3594469b2afa609295ec271: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:13.137046 4694 log.cc:824] T 00000000000000000000000000000000 P 559866f5f3594469b2afa609295ec271: Log is configured to *not* fsync() on all Append() calls
I20260925 05:01:13.138324 4697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:13.127085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57812 (local address 127.30.247.62:34079)
0925 05:01:13.127990 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:13.128000 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:13.128038 (+ 38us) server_negotiation.cc:407] Connection header received
0925 05:01:13.129697 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:13.129718 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:13.130161 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:13.130551 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:13.131151 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:13.132373 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:13.133762 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:13.134436 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:13.134652 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:13.135780 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:13.135787 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:13.135806 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:01:13.136371 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:13.136441 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:01:13.136461 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:13.136634 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:13.136829 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 05:01:13.137104 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":334,"threads_started":1}
I20260925 05:01:13.140096 4694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 559866f5f3594469b2afa609295ec271: No bootstrap required, opened a new log
I20260925 05:01:13.142007 31708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:01:13.147500 4694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 559866f5f3594469b2afa609295ec271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559866f5f3594469b2afa609295ec271" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 34079 } }
I20260925 05:01:13.147842 4694 raft_consensus.cc:374] T 00000000000000000000000000000000 P 559866f5f3594469b2afa609295ec271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:13.147956 4694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 559866f5f3594469b2afa609295ec271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 559866f5f3594469b2afa609295ec271, State: Initialized, Role: FOLLOWER
I20260925 05:01:13.148470 4694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 559866f5f3594469b2afa609295ec271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559866f5f3594469b2afa609295ec271" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 34079 } }
I20260925 05:01:13.148777 4694 raft_consensus.cc:388] T 00000000000000000000000000000000 P 559866f5f3594469b2afa609295ec271 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:01:13.148924 4694 raft_consensus.cc:482] T 00000000000000000000000000000000 P 559866f5f3594469b2afa609295ec271 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:01:13.149091 4694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 559866f5f3594469b2afa609295ec271 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:13.151307 4694 raft_consensus.cc:504] T 00000000000000000000000000000000 P 559866f5f3594469b2afa609295ec271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559866f5f3594469b2afa609295ec271" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 34079 } }
I20260925 05:01:13.151770 4694 leader_election.cc:302] T 00000000000000000000000000000000 P 559866f5f3594469b2afa609295ec271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 559866f5f3594469b2afa609295ec271; no voters:
I20260925 05:01:13.152377 4694 leader_election.cc:288] T 00000000000000000000000000000000 P 559866f5f3594469b2afa609295ec271 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:01:13.152467 4699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 559866f5f3594469b2afa609295ec271 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:01:13.154479 4694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 559866f5f3594469b2afa609295ec271 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:01:13.154917 4699 raft_consensus.cc:686] T 00000000000000000000000000000000 P 559866f5f3594469b2afa609295ec271 [term 1 LEADER]: Becoming Leader. State: Replica: 559866f5f3594469b2afa609295ec271, State: Running, Role: LEADER
I20260925 05:01:13.155504 4699 consensus_queue.cc:237] T 00000000000000000000000000000000 P 559866f5f3594469b2afa609295ec271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559866f5f3594469b2afa609295ec271" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 34079 } }
I20260925 05:01:13.158855 4701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 559866f5f3594469b2afa609295ec271 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 559866f5f3594469b2afa609295ec271. Latest consensus state: current_term: 1 leader_uuid: "559866f5f3594469b2afa609295ec271" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559866f5f3594469b2afa609295ec271" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 34079 } } }
I20260925 05:01:13.159307 4701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 559866f5f3594469b2afa609295ec271 [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:13.159039 4700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 559866f5f3594469b2afa609295ec271 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "559866f5f3594469b2afa609295ec271" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559866f5f3594469b2afa609295ec271" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 34079 } } }
I20260925 05:01:13.159462 4700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 559866f5f3594469b2afa609295ec271 [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:13.160228 4710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:01:13.167896 4710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:01:13.176733 4710 catalog_manager.cc:1347] Generated new cluster ID: 3307a09c8962415880655c3b0137b7e0
I20260925 05:01:13.176860 4710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:01:13.195196 4710 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:01:13.196956 4710 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:01:13.205745 4710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 559866f5f3594469b2afa609295ec271: Generated new TSK 0
I20260925 05:01:13.206748 4710 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:01:13.228441 4697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:13.220484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57816 (local address 127.30.247.62:34079)
0925 05:01:13.220641 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:13.220646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:13.221064 (+ 418us) server_negotiation.cc:407] Connection header received
0925 05:01:13.221256 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:13.221262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:13.221353 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:13.221516 (+ 163us) server_negotiation.cc:226] Negotiated authn=JWT
0925 05:01:13.221920 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:13.223003 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:13.224321 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:13.225006 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:13.225121 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:13.225601 (+ 480us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 05:01:13.227834 (+ 2233us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44405/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 05:01:13.228139 (+ 305us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57816: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44405/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 05:01:13.228578 4697 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57816: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44405/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 05:01:13.230260 4723 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:13.220350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:34079 (local address 127.0.0.1:57816)
0925 05:01:13.220957 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:13.220972 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:13.221085 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:13.221525 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:13.221529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:13.221538 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 05:01:13.221778 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:13.221785 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:13.223128 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:13.223132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:13.224179 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:13.224189 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:13.225187 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:13.225191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:13.225322 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:13.225353 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 05:01:13.228062 (+ 2709us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44405/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 05:01:13.229856 (+ 1794us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.247.62:34079: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44405/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":1510,"thread_start_us":348,"threads_started":1}
I20260925 05:01:13.231590 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 4628
I20260925 05:01:13.232729 4689 generic_service.cc:196] Checking for leaks (request via RPC)
E20260925 05:01:13.243848 4724 webserver.cc:533] Webserver: sslize failed:
E20260925 05:01:13.243985 4724 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 05:01:13.324040 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 4628
2026-09-25T05:01:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (706 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T05:01:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:01:13Z Disabled control of system clock
I20260925 05:01:13.400449 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:36331
--webserver_interface=127.30.247.62
--webserver_port=0
--builtin_ntp_servers=127.30.247.20:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:36331
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312430481417-31708-0/testchainca.cert with env {}
W20260925 05:01:13.710139 4729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:13.710400 4729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:13.710489 4729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:13.717247 4729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:01:13.717334 4729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:13.717371 4729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:13.717424 4729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:01:13.717469 4729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 05:01:13.717521 4729 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 05:01:13.717561 4729 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 05:01:13.726634 4729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:36331
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:36331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312430481417-31708-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:13.727694 4729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:13.729164 4729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:13.741122 4734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:13.741165 4737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:13.741258 4735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:13.742386 4729 server_base.cc:1034] running on GCE node
I20260925 05:01:13.743078 4729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:13.744256 4729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:13.745487 4729 hybrid_clock.cc:648] HybridClock initialized: now 1790312473745437 us; error 37 us; skew 500 ppm
I20260925 05:01:13.745831 4729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:01:13.748095 4729 webserver.cc:460] Webserver started at http://127.30.247.62:40451/ using document root <none> and password file <none>
I20260925 05:01:13.748865 4729 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:13.749004 4729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:13.749297 4729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:13.751889 4729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "f6486cb3dbaa4e1081c2cf7602a6ef7c"
format_stamp: "Formatted at 2026-09-25 05:01:13 on dist-test-slave-5n86"
server_key: "769b8d74fa67f2aab38b9c3e8482c623"
server_key_iv: "95805f006e6bf959cbabf224bf459f5f"
server_key_version: "encryptionkey@0"
I20260925 05:01:13.752749 4729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "f6486cb3dbaa4e1081c2cf7602a6ef7c"
format_stamp: "Formatted at 2026-09-25 05:01:13 on dist-test-slave-5n86"
server_key: "769b8d74fa67f2aab38b9c3e8482c623"
server_key_iv: "95805f006e6bf959cbabf224bf459f5f"
server_key_version: "encryptionkey@0"
I20260925 05:01:13.757383 4729 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 05:01:13.760480 4743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:13.761751 4729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:01:13.761935 4729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "f6486cb3dbaa4e1081c2cf7602a6ef7c"
format_stamp: "Formatted at 2026-09-25 05:01:13 on dist-test-slave-5n86"
server_key: "769b8d74fa67f2aab38b9c3e8482c623"
server_key_iv: "95805f006e6bf959cbabf224bf459f5f"
server_key_version: "encryptionkey@0"
I20260925 05:01:13.762143 4729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:13.778607 4729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:13.779289 4729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:13.779568 4729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:13.797256 4729 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:36331
I20260925 05:01:13.797297 4794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:36331 every 8 connection(s)
I20260925 05:01:13.798484 4729 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:01:13.801918 4795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:13.806301 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 4729
I20260925 05:01:13.806504 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:01:13.807116 31708 external_mini_cluster.cc:1428] Setting key 5cb1a75ed04dd88099a1b614aea8ec09
I20260925 05:01:13.811998 4795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6486cb3dbaa4e1081c2cf7602a6ef7c: Bootstrap starting.
I20260925 05:01:13.815342 4795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6486cb3dbaa4e1081c2cf7602a6ef7c: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:13.816395 4795 log.cc:824] T 00000000000000000000000000000000 P f6486cb3dbaa4e1081c2cf7602a6ef7c: Log is configured to *not* fsync() on all Append() calls
I20260925 05:01:13.819484 4795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6486cb3dbaa4e1081c2cf7602a6ef7c: No bootstrap required, opened a new log
I20260925 05:01:13.820722 4798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:13.808886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.30.247.62:36331)
0925 05:01:13.809875 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:13.809888 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:13.809928 (+ 40us) server_negotiation.cc:407] Connection header received
0925 05:01:13.811501 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:13.811522 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:13.811936 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:13.812297 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:13.812834 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:13.814069 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:13.815883 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:13.816648 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:13.816772 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:13.818015 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:13.818027 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:13.818046 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:01:13.818760 (+ 714us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:13.818855 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:01:13.818885 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:13.819014 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:13.819215 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 05:01:13.819505 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":363,"threads_started":1}
I20260925 05:01:13.824404 31708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:01:13.827843 4795 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6486cb3dbaa4e1081c2cf7602a6ef7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6486cb3dbaa4e1081c2cf7602a6ef7c" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36331 } }
I20260925 05:01:13.828190 4795 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6486cb3dbaa4e1081c2cf7602a6ef7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:13.828298 4795 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6486cb3dbaa4e1081c2cf7602a6ef7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6486cb3dbaa4e1081c2cf7602a6ef7c, State: Initialized, Role: FOLLOWER
I20260925 05:01:13.828883 4795 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6486cb3dbaa4e1081c2cf7602a6ef7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6486cb3dbaa4e1081c2cf7602a6ef7c" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36331 } }
I20260925 05:01:13.829123 4795 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6486cb3dbaa4e1081c2cf7602a6ef7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:01:13.829268 4795 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6486cb3dbaa4e1081c2cf7602a6ef7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:01:13.829454 4795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6486cb3dbaa4e1081c2cf7602a6ef7c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:13.831697 4795 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6486cb3dbaa4e1081c2cf7602a6ef7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6486cb3dbaa4e1081c2cf7602a6ef7c" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36331 } }
I20260925 05:01:13.832171 4795 leader_election.cc:302] T 00000000000000000000000000000000 P f6486cb3dbaa4e1081c2cf7602a6ef7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6486cb3dbaa4e1081c2cf7602a6ef7c; no voters:
I20260925 05:01:13.832780 4795 leader_election.cc:288] T 00000000000000000000000000000000 P f6486cb3dbaa4e1081c2cf7602a6ef7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:01:13.832908 4800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6486cb3dbaa4e1081c2cf7602a6ef7c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:01:13.833571 4800 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6486cb3dbaa4e1081c2cf7602a6ef7c [term 1 LEADER]: Becoming Leader. State: Replica: f6486cb3dbaa4e1081c2cf7602a6ef7c, State: Running, Role: LEADER
I20260925 05:01:13.834084 4800 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6486cb3dbaa4e1081c2cf7602a6ef7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6486cb3dbaa4e1081c2cf7602a6ef7c" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36331 } }
I20260925 05:01:13.834780 4795 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6486cb3dbaa4e1081c2cf7602a6ef7c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:01:13.836715 4802 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6486cb3dbaa4e1081c2cf7602a6ef7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6486cb3dbaa4e1081c2cf7602a6ef7c. Latest consensus state: current_term: 1 leader_uuid: "f6486cb3dbaa4e1081c2cf7602a6ef7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6486cb3dbaa4e1081c2cf7602a6ef7c" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36331 } } }
I20260925 05:01:13.836844 4801 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6486cb3dbaa4e1081c2cf7602a6ef7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6486cb3dbaa4e1081c2cf7602a6ef7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6486cb3dbaa4e1081c2cf7602a6ef7c" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36331 } } }
I20260925 05:01:13.837278 4802 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6486cb3dbaa4e1081c2cf7602a6ef7c [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:13.837343 4801 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6486cb3dbaa4e1081c2cf7602a6ef7c [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:13.839290 4807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:01:13.843922 4807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:01:13.852813 4807 catalog_manager.cc:1347] Generated new cluster ID: 9bf7ce71e90a44399c52da6171c9435d
I20260925 05:01:13.852923 4807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:01:13.878759 4807 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:01:13.880103 4807 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:01:13.887257 4807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6486cb3dbaa4e1081c2cf7602a6ef7c: Generated new TSK 0
I20260925 05:01:13.888077 4807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:01:13.995786 4798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:13.909571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34124 (local address 127.30.247.62:36331)
0925 05:01:13.909727 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:13.909732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:13.910124 (+ 392us) server_negotiation.cc:407] Connection header received
0925 05:01:13.910385 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:13.910389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:13.910459 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:13.910585 (+ 126us) server_negotiation.cc:226] Negotiated authn=JWT
0925 05:01:13.911108 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:13.911820 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:13.913854 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:13.914325 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:13.914417 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:13.914937 (+ 520us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 05:01:13.995184 (+ 80247us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 05:01:13.995529 (+ 345us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34124: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 05:01:13.995977 4798 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34124: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 05:01:13.996212 4824 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:13.909406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36331 (local address 127.0.0.1:34124)
0925 05:01:13.910017 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:13.910037 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:13.910179 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:13.910631 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:13.910636 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:13.910652 (+ 16us) client_negotiation.cc:189] Negotiated authn=JWT
0925 05:01:13.910961 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:13.910973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:13.911963 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:13.911969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:13.913707 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:13.913721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:13.914486 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:13.914493 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:13.914667 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:13.914711 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 05:01:13.995490 (+ 80779us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 05:01:13.995819 (+ 329us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.247.62:36331: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":317,"threads_started":1}
I20260925 05:01:13.997640 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 4729
I20260925 05:01:13.998807 4790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:14.099615 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 4729
2026-09-25T05:01:14Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (757 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (397 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (392 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:01:14 dist-test-slave-5n86 krb5kdc[4839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:01:14 dist-test-slave-5n86 krb5kdc[4839](info): set up 2 sockets
Sep 25 05:01:14 dist-test-slave-5n86 krb5kdc[4839](info): commencing operation
krb5kdc: starting...
W20260925 05:01:15.981376 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.006s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:01:16 dist-test-slave-5n86 krb5kdc[4839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:01:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:01:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:01:16.185139 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:40981
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:40981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:16.499961 4852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:16.500229 4852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:16.500394 4852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:16.507596 4852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:01:16.507699 4852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:16.507786 4852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:16.507891 4852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:01:16.507987 4852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:01:16.517547 4852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:40981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:40981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:16.518787 4852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:16.520056 4852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:16.531921 4858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:16.531921 4857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:16.531957 4860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:16.532809 4852 server_base.cc:1034] running on GCE node
I20260925 05:01:16.533540 4852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:16.534832 4852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:16.536079 4852 hybrid_clock.cc:648] HybridClock initialized: now 1790312476536020 us; error 49 us; skew 500 ppm
I20260925 05:01:16.536438 4852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:16 dist-test-slave-5n86 krb5kdc[4839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312476, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:16.539580 4852 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:01:16.541117 4852 webserver.cc:460] Webserver started at http://127.30.247.62:33767/ using document root <none> and password file <none>
I20260925 05:01:16.541790 4852 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:16.541888 4852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:16.542220 4852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:16.545644 4852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "430aa25dcf2344c995844070ca52e5f4"
format_stamp: "Formatted at 2026-09-25 05:01:16 on dist-test-slave-5n86"
server_key: "7ee82fa5b53838d0f09040971de850ea"
server_key_iv: "5100a28c65311ce8daca2efa3fbdb66a"
server_key_version: "encryptionkey@0"
I20260925 05:01:16.546497 4852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "430aa25dcf2344c995844070ca52e5f4"
format_stamp: "Formatted at 2026-09-25 05:01:16 on dist-test-slave-5n86"
server_key: "7ee82fa5b53838d0f09040971de850ea"
server_key_iv: "5100a28c65311ce8daca2efa3fbdb66a"
server_key_version: "encryptionkey@0"
I20260925 05:01:16.551817 4852 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 05:01:16.555444 4867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:16.556838 4852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 05:01:16.557013 4852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "430aa25dcf2344c995844070ca52e5f4"
format_stamp: "Formatted at 2026-09-25 05:01:16 on dist-test-slave-5n86"
server_key: "7ee82fa5b53838d0f09040971de850ea"
server_key_iv: "5100a28c65311ce8daca2efa3fbdb66a"
server_key_version: "encryptionkey@0"
I20260925 05:01:16.557204 4852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:16.595399 4852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:16.599519 4852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:16.599819 4852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:16.618067 4852 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:40981
I20260925 05:01:16.618161 4918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:40981 every 8 connection(s)
I20260925 05:01:16.619386 4852 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:01:16.622915 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 4852
I20260925 05:01:16.623124 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:01:16.623260 4919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:16.623948 31708 external_mini_cluster.cc:1428] Setting key 54c2058f9f1212fadaba6abd37c27ac0
I20260925 05:01:16.636480 4919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 430aa25dcf2344c995844070ca52e5f4: Bootstrap starting.
Sep 25 05:01:16 dist-test-slave-5n86 krb5kdc[4839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:01:16.639876 4919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 430aa25dcf2344c995844070ca52e5f4: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:16.641238 4919 log.cc:824] T 00000000000000000000000000000000 P 430aa25dcf2344c995844070ca52e5f4: Log is configured to *not* fsync() on all Append() calls
I20260925 05:01:16.644212 4919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 430aa25dcf2344c995844070ca52e5f4: No bootstrap required, opened a new log
I20260925 05:01:16.646827 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:16.625977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38218 (local address 127.30.247.62:40981)
0925 05:01:16.626793 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:16.626806 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:16.626846 (+ 40us) server_negotiation.cc:407] Connection header received
0925 05:01:16.628336 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:16.628355 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:16.628840 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:16.629173 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:16.630848 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:16.632425 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:16.633789 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:16.634527 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:16.634729 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:16.638900 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:16.638910 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:16.638930 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:16.638986 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:16.642626 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:16.643274 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:16.643281 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:16.643290 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:16.643495 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:16.643843 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:16.643847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:16.643850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:16.644491 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:16.644848 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:16.645119 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 05:01:16.645503 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":411,"threads_started":1}
I20260925 05:01:16.652993 4919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 430aa25dcf2344c995844070ca52e5f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "430aa25dcf2344c995844070ca52e5f4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40981 } }
I20260925 05:01:16.653306 4919 raft_consensus.cc:374] T 00000000000000000000000000000000 P 430aa25dcf2344c995844070ca52e5f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:16.653367 4919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 430aa25dcf2344c995844070ca52e5f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 430aa25dcf2344c995844070ca52e5f4, State: Initialized, Role: FOLLOWER
I20260925 05:01:16.653878 4919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 430aa25dcf2344c995844070ca52e5f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "430aa25dcf2344c995844070ca52e5f4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40981 } }
I20260925 05:01:16.654075 4919 raft_consensus.cc:388] T 00000000000000000000000000000000 P 430aa25dcf2344c995844070ca52e5f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:01:16.654219 4919 raft_consensus.cc:482] T 00000000000000000000000000000000 P 430aa25dcf2344c995844070ca52e5f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:01:16.654335 4919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 430aa25dcf2344c995844070ca52e5f4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:16.656687 4919 raft_consensus.cc:504] T 00000000000000000000000000000000 P 430aa25dcf2344c995844070ca52e5f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "430aa25dcf2344c995844070ca52e5f4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40981 } }
I20260925 05:01:16.657101 4919 leader_election.cc:302] T 00000000000000000000000000000000 P 430aa25dcf2344c995844070ca52e5f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 430aa25dcf2344c995844070ca52e5f4; no voters:
I20260925 05:01:16.657595 4919 leader_election.cc:288] T 00000000000000000000000000000000 P 430aa25dcf2344c995844070ca52e5f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:01:16.657742 4924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 430aa25dcf2344c995844070ca52e5f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:01:16.658388 4924 raft_consensus.cc:686] T 00000000000000000000000000000000 P 430aa25dcf2344c995844070ca52e5f4 [term 1 LEADER]: Becoming Leader. State: Replica: 430aa25dcf2344c995844070ca52e5f4, State: Running, Role: LEADER
I20260925 05:01:16.658905 4924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 430aa25dcf2344c995844070ca52e5f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "430aa25dcf2344c995844070ca52e5f4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40981 } }
I20260925 05:01:16.659550 4919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 430aa25dcf2344c995844070ca52e5f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:01:16.661957 4926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 430aa25dcf2344c995844070ca52e5f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 430aa25dcf2344c995844070ca52e5f4. Latest consensus state: current_term: 1 leader_uuid: "430aa25dcf2344c995844070ca52e5f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "430aa25dcf2344c995844070ca52e5f4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40981 } } }
I20260925 05:01:16.662509 4926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 430aa25dcf2344c995844070ca52e5f4 [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:16.662048 4925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 430aa25dcf2344c995844070ca52e5f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "430aa25dcf2344c995844070ca52e5f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "430aa25dcf2344c995844070ca52e5f4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40981 } } }
I20260925 05:01:16.663275 4925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 430aa25dcf2344c995844070ca52e5f4 [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:16.663373 4933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:01:16.668488 4933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:01:16.677209 4933 catalog_manager.cc:1347] Generated new cluster ID: 6cbb8f4a8ad943f1b2a82ccee8f14511
I20260925 05:01:16.677322 4933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:01:16.695276 4933 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:01:16.696609 4933 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:01:16.707661 4933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 430aa25dcf2344c995844070ca52e5f4: Generated new TSK 0
I20260925 05:01:16.708679 4933 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
I20260925 05:01:16.756805 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:0
--local_ip_for_outbound_sockets=127.30.247.1
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:40981
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:17.088617 4945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:17.088915 4945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:17.089069 4945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:17.096191 4945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:17.096287 4945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:17.096457 4945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:01:17.106020 4945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:40981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:17.107215 4945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:17.108470 4945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:17.121361 4953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:17.121501 4950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:17.121475 4951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:17.122574 4945 server_base.cc:1034] running on GCE node
I20260925 05:01:17.123157 4945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:17.124128 4945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:17.125344 4945 hybrid_clock.cc:648] HybridClock initialized: now 1790312477125295 us; error 48 us; skew 500 ppm
I20260925 05:01:17.125639 4945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:17 dist-test-slave-5n86 krb5kdc[4839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312477, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:17.128924 4945 init.cc:377] Logged in from keytab as kudu/127.30.247.1@KRBTEST.COM (short username kudu)
I20260925 05:01:17.130956 4945 webserver.cc:460] Webserver started at http://127.30.247.1:41839/ using document root <none> and password file <none>
I20260925 05:01:17.131589 4945 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:17.131732 4945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:17.132030 4945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:17.134881 4945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-0/data/instance:
uuid: "2130a53ec5a041488b3283efeef7b58e"
format_stamp: "Formatted at 2026-09-25 05:01:17 on dist-test-slave-5n86"
server_key: "b0aeb361e727bca96785a2316b9262f0"
server_key_iv: "1dc37ec93085e6d759e23e41b716efd1"
server_key_version: "encryptionkey@0"
I20260925 05:01:17.135604 4945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance:
uuid: "2130a53ec5a041488b3283efeef7b58e"
format_stamp: "Formatted at 2026-09-25 05:01:17 on dist-test-slave-5n86"
server_key: "b0aeb361e727bca96785a2316b9262f0"
server_key_iv: "1dc37ec93085e6d759e23e41b716efd1"
server_key_version: "encryptionkey@0"
I20260925 05:01:17.140311 4945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260925 05:01:17.143589 4960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:17.144877 4945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:01:17.145040 4945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "2130a53ec5a041488b3283efeef7b58e"
format_stamp: "Formatted at 2026-09-25 05:01:17 on dist-test-slave-5n86"
server_key: "b0aeb361e727bca96785a2316b9262f0"
server_key_iv: "1dc37ec93085e6d759e23e41b716efd1"
server_key_version: "encryptionkey@0"
I20260925 05:01:17.145237 4945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:17.159804 4945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:17.163931 4945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:17.164199 4945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:17.165194 4945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:17.166399 4945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:17.166501 4945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:17.166640 4945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:17.166710 4945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:17.200110 4945 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:42313
I20260925 05:01:17.200153 5072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:42313 every 8 connection(s)
I20260925 05:01:17.201411 4945 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:01:17.205435 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 4945
I20260925 05:01:17.205631 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance
I20260925 05:01:17.206269 31708 external_mini_cluster.cc:1428] Setting key 9a84994bcd0d96834daf881b41b848da
Sep 25 05:01:17 dist-test-slave-5n86 krb5kdc[4839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312477, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:01:17.221386 5075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:17.203902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:35039 (local address 127.30.247.62:40981)
0925 05:01:17.204301 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:17.204306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:17.205405 (+ 1099us) server_negotiation.cc:407] Connection header received
0925 05:01:17.207632 (+ 2227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:17.207639 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:17.207978 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:17.208100 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:17.209607 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:17.210344 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:17.211828 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:17.212238 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:17.212370 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:17.215808 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:17.215819 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:17.215827 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:17.215857 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:17.218438 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:17.219133 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:17.219139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:17.219147 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:17.219254 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:17.219668 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:17.219673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:17.219676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:17.219943 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:17.220095 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:17.220829 (+ 734us) server_negotiation.cc:299] Negotiation successful
0925 05:01:17.221031 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":198,"threads_started":1}
I20260925 05:01:17.222299 5076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:17.204456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40981 (local address 127.30.247.1:35039)
0925 05:01:17.205246 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:17.205281 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:17.207269 (+ 1988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:17.208345 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:17.208354 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:17.208916 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:17.209447 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:17.209463 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:17.210494 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:17.210500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:17.211683 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:17.211693 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:17.212459 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:17.212465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:17.212748 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:17.213425 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:17.213445 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:17.215599 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:17.218581 (+ 2982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:17.218589 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:17.218605 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:17.219006 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:17.219364 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:17.219370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:17.219374 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:17.219550 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:17.220069 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:17.220076 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:17.220405 (+ 329us) client_negotiation.cc:769] Sending connection context
0925 05:01:17.220705 (+ 300us) client_negotiation.cc:240] Negotiation successful
0925 05:01:17.221204 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":272,"threads_started":1}
I20260925 05:01:17.223970 5073 heartbeater.cc:347] Connected to a master server at 127.30.247.62:40981
I20260925 05:01:17.224294 5073 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.30.247.2@KRBTEST.COM; defaulting to no policy
I20260925 05:01:17.225163 5073 heartbeater.cc:511] Master 127.30.247.62:40981 requested a full tablet report, sending...
I20260925 05:01:17.227260 4884 ts_manager.cc:194] Registered new tserver with Master: 2130a53ec5a041488b3283efeef7b58e (127.30.247.1:42313)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.2@KRBTEST.COM" created.
I20260925 05:01:17.228873 4884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.1@KRBTEST.COM'} at 127.30.247.1:35039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
I20260925 05:01:17.251713 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:0
--local_ip_for_outbound_sockets=127.30.247.2
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:40981
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:17.561546 5079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:17.561856 5079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:17.562023 5079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:17.569499 5079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:17.569598 5079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:17.569772 5079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:01:17.579288 5079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:40981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:17.580579 5079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:17.581853 5079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:17.594605 5084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:17.594551 5087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:17.594512 5085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:17.595319 5079 server_base.cc:1034] running on GCE node
I20260925 05:01:17.595893 5079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:17.596853 5079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:17.598076 5079 hybrid_clock.cc:648] HybridClock initialized: now 1790312477598036 us; error 33 us; skew 500 ppm
I20260925 05:01:17.598438 5079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:17 dist-test-slave-5n86 krb5kdc[4839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312477, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:17.606953 5079 init.cc:377] Logged in from keytab as kudu/127.30.247.2@KRBTEST.COM (short username kudu)
I20260925 05:01:17.608317 5079 webserver.cc:460] Webserver started at http://127.30.247.2:32795/ using document root <none> and password file <none>
I20260925 05:01:17.609026 5079 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:17.609125 5079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:17.609380 5079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:17.612048 5079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-1/data/instance:
uuid: "76b3e8e316214c9cae4a78d6b06dfbc7"
format_stamp: "Formatted at 2026-09-25 05:01:17 on dist-test-slave-5n86"
server_key: "22e74f6f997c12fe6e7a9e358b15e810"
server_key_iv: "0970495848f4d5c753af1315781dfac8"
server_key_version: "encryptionkey@0"
I20260925 05:01:17.612821 5079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance:
uuid: "76b3e8e316214c9cae4a78d6b06dfbc7"
format_stamp: "Formatted at 2026-09-25 05:01:17 on dist-test-slave-5n86"
server_key: "22e74f6f997c12fe6e7a9e358b15e810"
server_key_iv: "0970495848f4d5c753af1315781dfac8"
server_key_version: "encryptionkey@0"
I20260925 05:01:17.617738 5079 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 05:01:17.621136 5094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:17.622483 5079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 05:01:17.622669 5079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "76b3e8e316214c9cae4a78d6b06dfbc7"
format_stamp: "Formatted at 2026-09-25 05:01:17 on dist-test-slave-5n86"
server_key: "22e74f6f997c12fe6e7a9e358b15e810"
server_key_iv: "0970495848f4d5c753af1315781dfac8"
server_key_version: "encryptionkey@0"
I20260925 05:01:17.622915 5079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:17.641795 5079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:17.645625 5079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:17.645907 5079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:17.646817 5079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:17.648106 5079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:17.648206 5079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:17.648365 5079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:17.648447 5079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:17.680938 5079 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:42241
I20260925 05:01:17.680974 5206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:42241 every 8 connection(s)
I20260925 05:01:17.682137 5079 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:01:17.689281 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 5079
I20260925 05:01:17.689474 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance
I20260925 05:01:17.690085 31708 external_mini_cluster.cc:1428] Setting key 08cd6545b35638d44450b41fa13fc23a
Sep 25 05:01:17 dist-test-slave-5n86 krb5kdc[4839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312477, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:01:17.704071 5075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:17.684803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:38687 (local address 127.30.247.62:40981)
0925 05:01:17.685010 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:17.685016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:17.686217 (+ 1201us) server_negotiation.cc:407] Connection header received
0925 05:01:17.688085 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:17.688090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:17.688155 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:17.688247 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:17.690482 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:17.691294 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:17.693666 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:17.694088 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:17.694191 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:17.698417 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:17.698422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:17.698425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:17.698453 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:17.701174 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:17.701881 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:17.701886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:17.701888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:17.702010 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:17.702438 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:17.702447 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:17.702449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:17.702687 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:17.702857 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:17.703515 (+ 658us) server_negotiation.cc:299] Negotiation successful
0925 05:01:17.703725 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 05:01:17.705008 5209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:17.685311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40981 (local address 127.30.247.2:38687)
0925 05:01:17.686050 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:17.686098 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:17.687813 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:17.688471 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:17.688479 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:17.689428 (+ 949us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:17.690265 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:17.690306 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:17.691454 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:17.691458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:17.693505 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:17.693524 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:17.694344 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:17.694354 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:17.694706 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:17.695430 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:17.695471 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:17.698176 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:17.701312 (+ 3136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:17.701321 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:17.701344 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:17.701735 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:17.702125 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:17.702129 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:17.702133 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:17.702306 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:17.702817 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:17.702825 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:17.703167 (+ 342us) client_negotiation.cc:769] Sending connection context
0925 05:01:17.703424 (+ 257us) client_negotiation.cc:240] Negotiation successful
0925 05:01:17.703912 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":301,"threads_started":1}
I20260925 05:01:17.706375 5207 heartbeater.cc:347] Connected to a master server at 127.30.247.62:40981
I20260925 05:01:17.706691 5207 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:17.707564 5207 heartbeater.cc:511] Master 127.30.247.62:40981 requested a full tablet report, sending...
I20260925 05:01:17.709189 4884 ts_manager.cc:194] Registered new tserver with Master: 76b3e8e316214c9cae4a78d6b06dfbc7 (127.30.247.2:42241)
I20260925 05:01:17.709959 4884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.2@KRBTEST.COM'} at 127.30.247.2:38687
WARNING: no policy specified for kudu/127.30.247.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
I20260925 05:01:17.735831 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:0
--local_ip_for_outbound_sockets=127.30.247.3
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:40981
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:18.054989 5212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:18.055280 5212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:18.055456 5212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:18.062635 5212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:18.062775 5212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:18.062999 5212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:01:18.072233 5212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:40981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:18.073488 5212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:18.074760 5212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:18.087306 5220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:18.087349 5218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:18.087568 5217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:18.088708 5212 server_base.cc:1034] running on GCE node
I20260925 05:01:18.089319 5212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:18.090191 5212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:18.091379 5212 hybrid_clock.cc:648] HybridClock initialized: now 1790312478091334 us; error 43 us; skew 500 ppm
I20260925 05:01:18.091672 5212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:18 dist-test-slave-5n86 krb5kdc[4839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312478, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:18.094712 5212 init.cc:377] Logged in from keytab as kudu/127.30.247.3@KRBTEST.COM (short username kudu)
I20260925 05:01:18.096082 5212 webserver.cc:460] Webserver started at http://127.30.247.3:42673/ using document root <none> and password file <none>
I20260925 05:01:18.096789 5212 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:18.096949 5212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:18.097244 5212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:18.099952 5212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-2/data/instance:
uuid: "e11959091f59472d92c1cd47d9564395"
format_stamp: "Formatted at 2026-09-25 05:01:18 on dist-test-slave-5n86"
server_key: "6bcb1e41efb36b1833a279491a902109"
server_key_iv: "f854af169c11d504fa5148a8efebff4f"
server_key_version: "encryptionkey@0"
I20260925 05:01:18.100744 5212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance:
uuid: "e11959091f59472d92c1cd47d9564395"
format_stamp: "Formatted at 2026-09-25 05:01:18 on dist-test-slave-5n86"
server_key: "6bcb1e41efb36b1833a279491a902109"
server_key_iv: "f854af169c11d504fa5148a8efebff4f"
server_key_version: "encryptionkey@0"
I20260925 05:01:18.105533 5212 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 05:01:18.108704 5227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:18.109959 5212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 05:01:18.110126 5212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "e11959091f59472d92c1cd47d9564395"
format_stamp: "Formatted at 2026-09-25 05:01:18 on dist-test-slave-5n86"
server_key: "6bcb1e41efb36b1833a279491a902109"
server_key_iv: "f854af169c11d504fa5148a8efebff4f"
server_key_version: "encryptionkey@0"
I20260925 05:01:18.110316 5212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:18.135417 5212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:18.139636 5212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:18.139921 5212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:18.140839 5212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:18.142073 5212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:18.142175 5212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:18.142323 5212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:18.142398 5212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:18.176590 5212 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:45235
I20260925 05:01:18.176659 5339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:45235 every 8 connection(s)
I20260925 05:01:18.177819 5212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:01:18.183367 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 5212
I20260925 05:01:18.183555 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance
I20260925 05:01:18.184032 31708 external_mini_cluster.cc:1428] Setting key 41e1346bc59941321988536330ba0b23
Sep 25 05:01:18 dist-test-slave-5n86 krb5kdc[4839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312478, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:01:18.197444 5075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:18.180275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:48759 (local address 127.30.247.62:40981)
0925 05:01:18.180466 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:18.180471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:18.181658 (+ 1187us) server_negotiation.cc:407] Connection header received
0925 05:01:18.183408 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:18.183413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:18.183522 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:18.183626 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:18.185437 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:18.186433 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:18.188075 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:18.188495 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:18.188682 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:18.192024 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:18.192028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:18.192031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:18.192061 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:18.194600 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:18.195253 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:18.195256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:18.195259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:18.195368 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:18.195784 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:18.195787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:18.195789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:18.196084 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:18.196203 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:18.196947 (+ 744us) server_negotiation.cc:299] Negotiation successful
0925 05:01:18.197102 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 05:01:18.198436 5342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:18.180778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40981 (local address 127.30.247.3:48759)
0925 05:01:18.181494 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:18.181530 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:18.183129 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:18.183874 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:18.183883 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:18.184416 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:18.185208 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:18.185254 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:18.186588 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:18.186594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:18.187945 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:18.187961 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:18.188743 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:18.188748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:18.188986 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:18.189680 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:18.189708 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:18.191824 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:18.194733 (+ 2909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:18.194740 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:18.194751 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:18.195153 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:18.195487 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:18.195490 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:18.195493 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:18.195683 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:18.196244 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:18.196251 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:18.196616 (+ 365us) client_negotiation.cc:769] Sending connection context
0925 05:01:18.196866 (+ 250us) client_negotiation.cc:240] Negotiation successful
0925 05:01:18.197218 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":306,"threads_started":1}
I20260925 05:01:18.199856 5340 heartbeater.cc:347] Connected to a master server at 127.30.247.62:40981
I20260925 05:01:18.200165 5340 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:18.200905 5340 heartbeater.cc:511] Master 127.30.247.62:40981 requested a full tablet report, sending...
I20260925 05:01:18.202252 4884 ts_manager.cc:194] Registered new tserver with Master: e11959091f59472d92c1cd47d9564395 (127.30.247.3:45235)
I20260925 05:01:18.203038 4884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.3@KRBTEST.COM'} at 127.30.247.3:48759
I20260925 05:01:18.211098 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:01:18.232506 5073 heartbeater.cc:503] Master 127.30.247.62:40981 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/test-admin.keytab.
Sep 25 05:01:18 dist-test-slave-5n86 krb5kdc[4839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:18.238140 31708 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 05:01:18.243067 5347 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:18.240767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50918 (local address 127.30.247.1:42313)
0925 05:01:18.241327 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:18.241334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:18.241361 (+ 27us) server_negotiation.cc:407] Connection header received
0925 05:01:18.241413 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:18.241425 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:18.241628 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:18.241751 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:18.242629 (+ 878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50918: BlockingRecv error: recv got EOF from 127.0.0.1:50918 (error 108)
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":293,"threads_started":1}
W20260925 05:01:18.243005 31708 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/krb5cc' of type FILE
Sep 25 05:01:18 dist-test-slave-5n86 krb5kdc[4839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:18.244622 31708 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 05:01:18 dist-test-slave-5n86 krb5kdc[4839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:18.246068 31708 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 05:01:18 dist-test-slave-5n86 krb5kdc[4839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.1@KRBTEST.COM
I20260925 05:01:18.261911 5347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:18.247691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50924 (local address 127.30.247.1:42313)
0925 05:01:18.247870 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:18.247876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:18.248005 (+ 129us) server_negotiation.cc:407] Connection header received
0925 05:01:18.248207 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:18.248212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:18.248281 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:18.248379 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:18.249422 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:18.250357 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:18.251591 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:18.252529 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:18.252699 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:18.256261 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:18.256275 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:18.256295 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:18.256323 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:18.259104 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:18.259800 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:18.259807 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:18.259809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:18.259920 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:18.260378 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:18.260385 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:18.260388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:18.260777 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:18.261071 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:18.261411 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 05:01:18.261671 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 05:01:18.267720 5347 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:18.266023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50928 (local address 127.30.247.1:42313)
0925 05:01:18.266191 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:18.266194 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:18.266590 (+ 396us) server_negotiation.cc:407] Connection header received
0925 05:01:18.266703 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:18.266706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:18.266769 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:18.266843 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:18.267540 (+ 697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50928: BlockingRecv error: recv got EOF from 127.0.0.1:50928 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 05:01:18.267946 31708 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/krb5cc' of type FILE
Sep 25 05:01:18 dist-test-slave-5n86 krb5kdc[4839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:18.269415 31708 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 05:01:18 dist-test-slave-5n86 krb5kdc[4839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:18.270658 31708 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 05:01:18 dist-test-slave-5n86 krb5kdc[4839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.1@KRBTEST.COM
I20260925 05:01:18.284174 5347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:18.271936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50934 (local address 127.30.247.1:42313)
0925 05:01:18.272134 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:18.272142 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:18.272326 (+ 184us) server_negotiation.cc:407] Connection header received
0925 05:01:18.272494 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:18.272501 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:18.272610 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:18.272710 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:18.273622 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:18.274297 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:18.275491 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:18.275904 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:18.276045 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:18.279459 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:18.279463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:18.279466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:18.279508 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:18.282106 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:18.282654 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:18.282657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:18.282659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:18.282764 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:18.283075 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:18.283078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:18.283079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:18.283298 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:18.283481 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:18.283727 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 05:01:18.283904 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260925 05:01:18.288012 5347 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:18.286423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50950 (local address 127.30.247.1:42313)
0925 05:01:18.286582 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:18.286585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:18.286912 (+ 327us) server_negotiation.cc:407] Connection header received
0925 05:01:18.286962 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:18.286965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:18.287038 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:18.287116 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:18.287755 (+ 639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50950: BlockingRecv error: recv got EOF from 127.0.0.1:50950 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 05:01:18.288164 31708 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312430481417-31708-0/krb5kdc/krb5cc' of type FILE
Sep 25 05:01:18 dist-test-slave-5n86 krb5kdc[4839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:18.289700 31708 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 05:01:18 dist-test-slave-5n86 krb5kdc[4839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:18.290916 31708 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 05:01:18 dist-test-slave-5n86 krb5kdc[4839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.1@KRBTEST.COM
I20260925 05:01:18.304181 5347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:18.292062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50964 (local address 127.30.247.1:42313)
0925 05:01:18.292271 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:18.292277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:18.292530 (+ 253us) server_negotiation.cc:407] Connection header received
0925 05:01:18.292688 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:18.292693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:18.292750 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:18.292851 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:18.293731 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:18.294417 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:18.295628 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:18.296027 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:18.296158 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:18.299390 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:18.299395 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:18.299398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:18.299426 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:18.301952 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:18.302616 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:18.302620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:18.302622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:18.302754 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:18.303107 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:18.303110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:18.303112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:18.303354 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:18.303563 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:18.303791 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 05:01:18.303935 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 05:01:18.305666 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 4945
I20260925 05:01:18.317082 5347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:18.306297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50980 (local address 127.30.247.1:42313)
0925 05:01:18.306444 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:18.306448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:18.306791 (+ 343us) server_negotiation.cc:407] Connection header received
0925 05:01:18.306902 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:18.306905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:18.306951 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:18.307025 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:18.307806 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:18.308464 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:18.309755 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:18.310181 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:18.310282 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:18.312236 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:18.312240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:18.312243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:18.312270 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:18.314719 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:18.315453 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:18.315457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:18.315459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:18.315591 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:18.315981 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:18.315984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:18.315986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:18.316207 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:18.316384 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:18.316701 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 05:01:18.316849 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:01:18.317739 5068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:18.399766 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 4945
I20260925 05:01:18.415762 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 5079
Sep 25 05:01:18 dist-test-slave-5n86 krb5kdc[4839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.2@KRBTEST.COM
I20260925 05:01:18.431097 5360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:18.416706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50846 (local address 127.30.247.2:42241)
0925 05:01:18.417290 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:18.417297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:18.417322 (+ 25us) server_negotiation.cc:407] Connection header received
0925 05:01:18.417383 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:18.417390 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:18.417590 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:18.417719 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:18.418630 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:18.419663 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:18.421019 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:18.421587 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:18.421689 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:18.425383 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:18.425393 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:18.425406 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:18.425445 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:18.428438 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:18.429135 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:18.429143 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:18.429145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:18.429262 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:18.429643 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:18.429647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:18.429649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:18.430000 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:18.430224 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:18.430504 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 05:01:18.430820 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":332,"threads_started":1}
I20260925 05:01:18.432392 5202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:18.512902 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 5079
I20260925 05:01:18.528393 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 5212
Sep 25 05:01:18 dist-test-slave-5n86 krb5kdc[4839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.3@KRBTEST.COM
I20260925 05:01:18.544283 5362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:18.529624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57896 (local address 127.30.247.3:45235)
0925 05:01:18.530341 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:18.530347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:18.530376 (+ 29us) server_negotiation.cc:407] Connection header received
0925 05:01:18.530432 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:18.530439 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:18.530649 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:18.530774 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:18.531754 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:18.532949 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:18.534154 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:18.534686 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:18.534787 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:18.538726 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:18.538736 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:18.538751 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:18.538784 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:18.541639 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:18.542317 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:18.542325 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:18.542327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:18.542460 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:18.542923 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:18.542928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:18.542931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:18.543280 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:18.543532 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:18.543813 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 05:01:18.544006 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":405,"threads_started":1}
I20260925 05:01:18.545632 5335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:18.626718 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 5212
I20260925 05:01:18.642059 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 4852
I20260925 05:01:18.643296 4914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:18.743937 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 4852
2026-09-25T05:01:18Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3859 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:01:18 dist-test-slave-5n86 krb5kdc[5367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:01:18 dist-test-slave-5n86 krb5kdc[5367](info): set up 2 sockets
Sep 25 05:01:18 dist-test-slave-5n86 krb5kdc[5367](info): commencing operation
krb5kdc: starting...
W20260925 05:01:19.829550 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:01:19 dist-test-slave-5n86 krb5kdc[5367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:01:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:01:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.62.keytab.
Entry for principal oryx/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.62.keytab.
I20260925 05:01:20.033185 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:44265
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.62.keytab
--principal=oryx/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:44265
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:20.345692 5380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:20.346064 5380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:20.346236 5380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:20.353259 5380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:01:20.353407 5380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:20.353523 5380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:20.353633 5380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:01:20.353725 5380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:01:20.363068 5380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:44265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.62.keytab
--principal=oryx/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:44265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:20.364370 5380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:20.365680 5380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:20.377368 5385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:20.377424 5386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:20.377452 5388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:20.378325 5380 server_base.cc:1034] running on GCE node
I20260925 05:01:20.379143 5380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:20.380523 5380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:20.381816 5380 hybrid_clock.cc:648] HybridClock initialized: now 1790312480381779 us; error 43 us; skew 500 ppm
I20260925 05:01:20.382161 5380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:20 dist-test-slave-5n86 krb5kdc[5367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312480, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:20.385298 5380 init.cc:377] Logged in from keytab as oryx/127.30.247.62@KRBTEST.COM (short username oryx)
I20260925 05:01:20.386761 5380 webserver.cc:460] Webserver started at http://127.30.247.62:37461/ using document root <none> and password file <none>
I20260925 05:01:20.387428 5380 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:20.387558 5380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:20.387858 5380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:20.390755 5380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "190a829b060c49298dcf7111c5dab8c6"
format_stamp: "Formatted at 2026-09-25 05:01:20 on dist-test-slave-5n86"
server_key: "77ba5e9132ff5b3b4a65444884f6f576"
server_key_iv: "0d2851b043f2d7cb5d972b4ad516c5d5"
server_key_version: "encryptionkey@0"
I20260925 05:01:20.391505 5380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "190a829b060c49298dcf7111c5dab8c6"
format_stamp: "Formatted at 2026-09-25 05:01:20 on dist-test-slave-5n86"
server_key: "77ba5e9132ff5b3b4a65444884f6f576"
server_key_iv: "0d2851b043f2d7cb5d972b4ad516c5d5"
server_key_version: "encryptionkey@0"
I20260925 05:01:20.396301 5380 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 05:01:20.399596 5395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:20.400888 5380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:01:20.401070 5380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "190a829b060c49298dcf7111c5dab8c6"
format_stamp: "Formatted at 2026-09-25 05:01:20 on dist-test-slave-5n86"
server_key: "77ba5e9132ff5b3b4a65444884f6f576"
server_key_iv: "0d2851b043f2d7cb5d972b4ad516c5d5"
server_key_version: "encryptionkey@0"
I20260925 05:01:20.401258 5380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:20.418280 5380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:20.422427 5380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:20.422700 5380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:20.440150 5380 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:44265
I20260925 05:01:20.440178 5446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:44265 every 8 connection(s)
I20260925 05:01:20.441428 5380 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:01:20.444907 5447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:20.450382 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 5380
I20260925 05:01:20.450592 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:01:20.451241 31708 external_mini_cluster.cc:1428] Setting key 5d9074bb18d57111604f6e62aedcdf5c
I20260925 05:01:20.455690 5447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6: Bootstrap starting.
I20260925 05:01:20.459388 5447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:20.460587 5447 log.cc:824] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6: Log is configured to *not* fsync() on all Append() calls
I20260925 05:01:20.463649 5447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6: No bootstrap required, opened a new log
Sep 25 05:01:20 dist-test-slave-5n86 krb5kdc[5367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM
I20260925 05:01:20.474076 5447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } }
I20260925 05:01:20.474511 5447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:20.474629 5447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 190a829b060c49298dcf7111c5dab8c6, State: Initialized, Role: FOLLOWER
I20260925 05:01:20.475265 5447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } }
I20260925 05:01:20.475528 5447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:01:20.475664 5447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:01:20.475864 5447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:20.475905 5450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:20.453173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49662 (local address 127.30.247.62:44265)
0925 05:01:20.453973 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:20.453987 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:20.454032 (+ 45us) server_negotiation.cc:407] Connection header received
0925 05:01:20.456327 (+ 2295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:20.456354 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:20.456838 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:20.457261 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:20.458495 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:20.460481 (+ 1986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:20.462126 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:20.463112 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:20.463289 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:20.467669 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:20.467682 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:20.467701 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:20.467748 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:20.471674 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:20.472324 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:20.472333 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:20.472342 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:20.472568 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:20.472907 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:20.472912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:20.472915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:20.473580 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:20.473865 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:20.474165 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 05:01:20.474600 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":357,"threads_started":1}
I20260925 05:01:20.478817 5447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } }
I20260925 05:01:20.479207 5447 leader_election.cc:302] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 190a829b060c49298dcf7111c5dab8c6; no voters:
I20260925 05:01:20.479717 5447 leader_election.cc:288] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:01:20.479854 5452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:01:20.480523 5452 raft_consensus.cc:686] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [term 1 LEADER]: Becoming Leader. State: Replica: 190a829b060c49298dcf7111c5dab8c6, State: Running, Role: LEADER
I20260925 05:01:20.481068 5452 consensus_queue.cc:237] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } }
I20260925 05:01:20.481709 5447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:01:20.484047 5453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "190a829b060c49298dcf7111c5dab8c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } }
I20260925 05:01:20.484160 5454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 190a829b060c49298dcf7111c5dab8c6. Latest consensus state: current_term: 1 leader_uuid: "190a829b060c49298dcf7111c5dab8c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } }
I20260925 05:01:20.484432 5453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:20.484501 5454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:20.486472 5459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:01:20.495271 5459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:01:20.502724 5459 catalog_manager.cc:1347] Generated new cluster ID: 69f5d87d0fd1476db9d81f6d4c7fdfba
I20260925 05:01:20.502837 5459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:01:20.511062 5459 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:01:20.512861 5459 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:01:20.529866 5459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6: Generated new TSK 0
I20260925 05:01:20.530821 5459 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.30.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.1.keytab.
Entry for principal oryx/127.30.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.1.keytab.
I20260925 05:01:20.588719 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:0
--local_ip_for_outbound_sockets=127.30.247.1
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:44265
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.1.keytab
--principal=oryx/127.30.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:20.906862 5473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:20.907109 5473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:20.907284 5473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:20.914347 5473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:20.914451 5473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:20.914623 5473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:01:20.924021 5473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.1.keytab
--principal=oryx/127.30.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:44265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:20.925228 5473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:20.926636 5473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:20.939250 5478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:20.939234 5479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:20.939260 5481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:20.939813 5473 server_base.cc:1034] running on GCE node
I20260925 05:01:20.940511 5473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:20.941468 5473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:20.942698 5473 hybrid_clock.cc:648] HybridClock initialized: now 1790312480942665 us; error 63 us; skew 500 ppm
I20260925 05:01:20.942998 5473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:20 dist-test-slave-5n86 krb5kdc[5367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312480, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:20.946261 5473 init.cc:377] Logged in from keytab as oryx/127.30.247.1@KRBTEST.COM (short username oryx)
I20260925 05:01:20.947654 5473 webserver.cc:460] Webserver started at http://127.30.247.1:39157/ using document root <none> and password file <none>
I20260925 05:01:20.948284 5473 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:20.948427 5473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:20.948781 5473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:20.951515 5473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/data/instance:
uuid: "41fed7f820404f16aa2d506396ef227c"
format_stamp: "Formatted at 2026-09-25 05:01:20 on dist-test-slave-5n86"
server_key: "57b18f70efd299f5c0c3c4cdc2a1fa5b"
server_key_iv: "7df987ca08ecea377f5d67910f9d3087"
server_key_version: "encryptionkey@0"
I20260925 05:01:20.952262 5473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance:
uuid: "41fed7f820404f16aa2d506396ef227c"
format_stamp: "Formatted at 2026-09-25 05:01:20 on dist-test-slave-5n86"
server_key: "57b18f70efd299f5c0c3c4cdc2a1fa5b"
server_key_iv: "7df987ca08ecea377f5d67910f9d3087"
server_key_version: "encryptionkey@0"
I20260925 05:01:20.957229 5473 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260925 05:01:20.960507 5488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:20.961855 5473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:01:20.962024 5473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "41fed7f820404f16aa2d506396ef227c"
format_stamp: "Formatted at 2026-09-25 05:01:20 on dist-test-slave-5n86"
server_key: "57b18f70efd299f5c0c3c4cdc2a1fa5b"
server_key_iv: "7df987ca08ecea377f5d67910f9d3087"
server_key_version: "encryptionkey@0"
I20260925 05:01:20.962209 5473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:20.988577 5473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:20.992465 5473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:20.992777 5473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:20.993697 5473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:20.994941 5473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:20.995051 5473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:20.995200 5473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:20.995275 5473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:21.027808 5473 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:33901
I20260925 05:01:21.027851 5600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:33901 every 8 connection(s)
I20260925 05:01:21.029201 5473 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:01:21.039281 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 5473
I20260925 05:01:21.039502 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance
I20260925 05:01:21.040257 31708 external_mini_cluster.cc:1428] Setting key 7d9ba55ac5f8b3dfeae9eee7e88bd071
Sep 25 05:01:21 dist-test-slave-5n86 krb5kdc[5367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312480, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM
I20260925 05:01:21.048599 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:21.031773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:37705 (local address 127.30.247.62:44265)
0925 05:01:21.032174 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:21.032178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:21.033219 (+ 1041us) server_negotiation.cc:407] Connection header received
0925 05:01:21.034980 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:21.034985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:21.035081 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:21.035200 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:21.036630 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:21.037332 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:21.038870 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:21.039277 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:21.039442 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:21.043189 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:21.043194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:21.043198 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:21.043226 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:21.045675 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:21.046358 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:21.046365 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:21.046373 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:21.046483 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:21.046907 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:21.046911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:21.046915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:21.047160 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:21.047316 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:21.047990 (+ 674us) server_negotiation.cc:299] Negotiation successful
0925 05:01:21.048204 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":221,"threads_started":1}
I20260925 05:01:21.049562 5604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:21.032356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.30.247.1:37705)
0925 05:01:21.033075 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:21.033111 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:21.034705 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:21.035424 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:21.035432 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:21.035899 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:21.036449 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:21.036465 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:21.037493 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:21.037500 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:21.038712 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:21.038727 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:21.039501 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:21.039507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:21.039758 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:21.040457 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:21.040477 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:21.042964 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:21.045813 (+ 2849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:21.045822 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:21.045838 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:21.046231 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:21.046599 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:21.046606 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:21.046609 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:21.046787 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:21.047288 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:21.047296 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:21.047620 (+ 324us) client_negotiation.cc:769] Sending connection context
0925 05:01:21.047874 (+ 254us) client_negotiation.cc:240] Negotiation successful
0925 05:01:21.048369 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":315,"threads_started":1}
I20260925 05:01:21.051047 5601 heartbeater.cc:347] Connected to a master server at 127.30.247.62:44265
I20260925 05:01:21.051339 5601 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:21.052012 5601 heartbeater.cc:511] Master 127.30.247.62:44265 requested a full tablet report, sending...
I20260925 05:01:21.054235 5412 ts_manager.cc:194] Registered new tserver with Master: 41fed7f820404f16aa2d506396ef227c (127.30.247.1:33901)
I20260925 05:01:21.056411 5412 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.247.1@KRBTEST.COM'} at 127.30.247.1:37705
WARNING: no policy specified for oryx/127.30.247.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.2.keytab.
Entry for principal oryx/127.30.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.2.keytab.
I20260925 05:01:21.088580 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:0
--local_ip_for_outbound_sockets=127.30.247.2
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:44265
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.2.keytab
--principal=oryx/127.30.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:21.399470 5607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:21.399717 5607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:21.399804 5607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:21.406903 5607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:21.406996 5607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:21.407102 5607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:01:21.416188 5607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.2.keytab
--principal=oryx/127.30.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:44265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:21.417330 5607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:21.418486 5607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:21.430964 5612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:21.431054 5615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:21.431016 5613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:21.431939 5607 server_base.cc:1034] running on GCE node
I20260925 05:01:21.432492 5607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:21.433387 5607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:21.434587 5607 hybrid_clock.cc:648] HybridClock initialized: now 1790312481434553 us; error 26 us; skew 500 ppm
I20260925 05:01:21.434896 5607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:21 dist-test-slave-5n86 krb5kdc[5367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312481, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:21.437990 5607 init.cc:377] Logged in from keytab as oryx/127.30.247.2@KRBTEST.COM (short username oryx)
I20260925 05:01:21.439520 5607 webserver.cc:460] Webserver started at http://127.30.247.2:35787/ using document root <none> and password file <none>
I20260925 05:01:21.440375 5607 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:21.440505 5607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:21.440817 5607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:21.444460 5607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/data/instance:
uuid: "87f2a171c1c8411e93504e0b8a47b426"
format_stamp: "Formatted at 2026-09-25 05:01:21 on dist-test-slave-5n86"
server_key: "e3ec09d5ddf6ec3391403317a4562254"
server_key_iv: "ff31bc61fa87c9da58cd4bc48a877b81"
server_key_version: "encryptionkey@0"
I20260925 05:01:21.445415 5607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance:
uuid: "87f2a171c1c8411e93504e0b8a47b426"
format_stamp: "Formatted at 2026-09-25 05:01:21 on dist-test-slave-5n86"
server_key: "e3ec09d5ddf6ec3391403317a4562254"
server_key_iv: "ff31bc61fa87c9da58cd4bc48a877b81"
server_key_version: "encryptionkey@0"
I20260925 05:01:21.451496 5607 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 05:01:21.455606 5622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:21.457158 5607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 05:01:21.457301 5607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "87f2a171c1c8411e93504e0b8a47b426"
format_stamp: "Formatted at 2026-09-25 05:01:21 on dist-test-slave-5n86"
server_key: "e3ec09d5ddf6ec3391403317a4562254"
server_key_iv: "ff31bc61fa87c9da58cd4bc48a877b81"
server_key_version: "encryptionkey@0"
I20260925 05:01:21.457501 5607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:21.472620 5607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:21.476922 5607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:21.477159 5607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:21.478226 5607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:21.479681 5607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:21.479756 5607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:21.479839 5607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:21.479887 5607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:21.513456 5607 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:43559
I20260925 05:01:21.513496 5734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:43559 every 8 connection(s)
I20260925 05:01:21.514703 5607 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:01:21.516000 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 5607
I20260925 05:01:21.516178 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance
I20260925 05:01:21.516763 31708 external_mini_cluster.cc:1428] Setting key c9c623fff7dcc619bb6a193d8e7c087e
Sep 25 05:01:21 dist-test-slave-5n86 krb5kdc[5367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312481, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM
I20260925 05:01:21.537719 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:21.517501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:49499 (local address 127.30.247.62:44265)
0925 05:01:21.517780 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:21.517786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:21.519142 (+ 1356us) server_negotiation.cc:407] Connection header received
0925 05:01:21.521138 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:21.521146 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:21.521232 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:21.521363 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:21.523085 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:21.524256 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:21.525911 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:21.526685 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:21.526829 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:21.530582 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:21.530591 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:21.530599 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:21.530647 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:21.534491 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:21.535234 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:21.535243 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:21.535250 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:21.535420 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:21.535840 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:21.535848 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:21.535852 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:21.536250 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:21.536407 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:21.537093 (+ 686us) server_negotiation.cc:299] Negotiation successful
0925 05:01:21.537329 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 05:01:21.538389 5737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:21.518170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.30.247.2:49499)
0925 05:01:21.518974 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:21.519021 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:21.520862 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:21.521575 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:21.521584 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:21.522097 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:21.522896 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:21.522919 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:21.524434 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:21.524442 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:21.525778 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:21.525790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:21.526907 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:21.526912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:21.527107 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:21.527788 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:21.527810 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:21.530322 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:21.534649 (+ 4327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:21.534655 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:21.534667 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:21.535083 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:21.535542 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:21.535545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:21.535547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:21.535718 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:21.536370 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:21.536375 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:21.536731 (+ 356us) client_negotiation.cc:769] Sending connection context
0925 05:01:21.536941 (+ 210us) client_negotiation.cc:240] Negotiation successful
0925 05:01:21.537304 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":358,"threads_started":1}
WARNING: no policy specified for oryx/127.30.247.3@KRBTEST.COM; defaulting to no policy
I20260925 05:01:21.539906 5735 heartbeater.cc:347] Connected to a master server at 127.30.247.62:44265
I20260925 05:01:21.540186 5735 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:21.541004 5735 heartbeater.cc:511] Master 127.30.247.62:44265 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.247.3@KRBTEST.COM" created.
I20260925 05:01:21.542735 5412 ts_manager.cc:194] Registered new tserver with Master: 87f2a171c1c8411e93504e0b8a47b426 (127.30.247.2:43559)
I20260925 05:01:21.544011 5412 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.247.2@KRBTEST.COM'} at 127.30.247.2:49499
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.3.keytab.
Entry for principal oryx/127.30.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.3.keytab.
I20260925 05:01:21.568315 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:0
--local_ip_for_outbound_sockets=127.30.247.3
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:44265
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.3.keytab
--principal=oryx/127.30.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:21.899569 5740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:21.899873 5740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:21.900048 5740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:21.907198 5740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:21.907336 5740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:21.907512 5740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:01:21.916788 5740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.3.keytab
--principal=oryx/127.30.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:44265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:21.917979 5740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:21.919226 5740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:21.932009 5748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:21.932068 5746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:21.932092 5745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:21.932753 5740 server_base.cc:1034] running on GCE node
I20260925 05:01:21.933293 5740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:21.934116 5740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:21.935329 5740 hybrid_clock.cc:648] HybridClock initialized: now 1790312481935297 us; error 42 us; skew 500 ppm
I20260925 05:01:21.935612 5740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:21 dist-test-slave-5n86 krb5kdc[5367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312481, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:21.938588 5740 init.cc:377] Logged in from keytab as oryx/127.30.247.3@KRBTEST.COM (short username oryx)
I20260925 05:01:21.939899 5740 webserver.cc:460] Webserver started at http://127.30.247.3:34051/ using document root <none> and password file <none>
I20260925 05:01:21.940485 5740 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:21.940667 5740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:21.940912 5740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:21.943554 5740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/data/instance:
uuid: "ad8603b082d240c287b1cf61856b67ce"
format_stamp: "Formatted at 2026-09-25 05:01:21 on dist-test-slave-5n86"
server_key: "eda2172d83462013c8036a341c088fe9"
server_key_iv: "23afb2dd63a196566dfbf41b16b12ca4"
server_key_version: "encryptionkey@0"
I20260925 05:01:21.944269 5740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance:
uuid: "ad8603b082d240c287b1cf61856b67ce"
format_stamp: "Formatted at 2026-09-25 05:01:21 on dist-test-slave-5n86"
server_key: "eda2172d83462013c8036a341c088fe9"
server_key_iv: "23afb2dd63a196566dfbf41b16b12ca4"
server_key_version: "encryptionkey@0"
I20260925 05:01:21.949366 5740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260925 05:01:21.952500 5755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:21.953781 5740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:01:21.953917 5740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "ad8603b082d240c287b1cf61856b67ce"
format_stamp: "Formatted at 2026-09-25 05:01:21 on dist-test-slave-5n86"
server_key: "eda2172d83462013c8036a341c088fe9"
server_key_iv: "23afb2dd63a196566dfbf41b16b12ca4"
server_key_version: "encryptionkey@0"
I20260925 05:01:21.954137 5740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:21.977662 5740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:21.981323 5740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:21.981650 5740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:21.982805 5740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:21.984184 5740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:21.984279 5740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:21.984398 5740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:21.984478 5740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:22.017129 5740 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:41837
I20260925 05:01:22.017225 5867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:41837 every 8 connection(s)
I20260925 05:01:22.018388 5740 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:01:22.027575 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 5740
I20260925 05:01:22.027738 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance
I20260925 05:01:22.028301 31708 external_mini_cluster.cc:1428] Setting key c7883d07a96c0a39e229401e3622a5c3
Sep 25 05:01:22 dist-test-slave-5n86 krb5kdc[5367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312481, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM
I20260925 05:01:22.038128 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:22.021038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:56229 (local address 127.30.247.62:44265)
0925 05:01:22.021215 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:22.021220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:22.022387 (+ 1167us) server_negotiation.cc:407] Connection header received
0925 05:01:22.024157 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:22.024165 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:22.024259 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:22.024381 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:22.025818 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.026588 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.028345 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.028860 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.029014 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:22.032897 (+ 3883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:22.032902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:22.032905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:22.032934 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:22.035286 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:22.035937 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:22.035945 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:22.035948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:22.036073 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:22.036479 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:22.036485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:22.036489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:22.036783 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:22.036971 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:22.037571 (+ 600us) server_negotiation.cc:299] Negotiation successful
0925 05:01:22.037776 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 05:01:22.039048 5870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:22.021468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.30.247.3:56229)
0925 05:01:22.022196 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:22.022243 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:22.023886 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:22.024633 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:22.024642 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:22.025118 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:22.025656 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:22.025671 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.026747 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.026755 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:22.028194 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:22.028206 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.029106 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.029113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:22.029396 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:22.030197 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:22.030230 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:22.032705 (+ 2475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:22.035419 (+ 2714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:22.035428 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:22.035442 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:22.035824 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:22.036189 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:22.036195 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:22.036199 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:22.036373 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:22.036949 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:22.036956 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:22.037248 (+ 292us) client_negotiation.cc:769] Sending connection context
0925 05:01:22.037501 (+ 253us) client_negotiation.cc:240] Negotiation successful
0925 05:01:22.037968 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":305,"threads_started":1}
I20260925 05:01:22.040318 5868 heartbeater.cc:347] Connected to a master server at 127.30.247.62:44265
I20260925 05:01:22.040668 5868 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:22.041325 5868 heartbeater.cc:511] Master 127.30.247.62:44265 requested a full tablet report, sending...
I20260925 05:01:22.042707 5412 ts_manager.cc:194] Registered new tserver with Master: ad8603b082d240c287b1cf61856b67ce (127.30.247.3:41837)
I20260925 05:01:22.043375 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:01:22.043643 5412 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.247.3@KRBTEST.COM'} at 127.30.247.3:56229
I20260925 05:01:22.059409 5601 heartbeater.cc:503] Master 127.30.247.62:44265 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.30.247.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.247.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.247.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.61.keytab.
Entry for principal oryx/127.30.247.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.61.keytab.
I20260925 05:01:22.088126 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.61:42081
--webserver_interface=127.30.247.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.61.keytab
--principal=oryx/127.30.247.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:44265,127.30.247.61:42081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:22.398918 5873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:22.399187 5873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:22.399361 5873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:22.406467 5873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:01:22.406558 5873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:22.406669 5873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:22.406756 5873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:01:22.406816 5873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:01:22.416064 5873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:44265,127.30.247.61:42081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.61.keytab
--principal=oryx/127.30.247.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.61:42081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.247.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
W20260925 05:01:22.416378 5873 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.30.247.62:44265,127.30.247.61:42081'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 05:01:22.417351 5873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:22.418610 5873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:22.429515 5878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:22.429538 5879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:22.429850 5881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:22.430413 5873 server_base.cc:1034] running on GCE node
I20260925 05:01:22.431082 5873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:22.432289 5873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:22.433494 5873 hybrid_clock.cc:648] HybridClock initialized: now 1790312482433407 us; error 63 us; skew 500 ppm
I20260925 05:01:22.433810 5873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:22 dist-test-slave-5n86 krb5kdc[5367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312482, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:22.436903 5873 init.cc:377] Logged in from keytab as oryx/127.30.247.61@KRBTEST.COM (short username oryx)
I20260925 05:01:22.438273 5873 webserver.cc:460] Webserver started at http://127.30.247.61:39823/ using document root <none> and password file <none>
I20260925 05:01:22.438921 5873 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:22.439050 5873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:22.439350 5873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:22.442754 5873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-1/data/instance:
uuid: "543dab666dc748caa9542055c3b85e43"
format_stamp: "Formatted at 2026-09-25 05:01:22 on dist-test-slave-5n86"
server_key: "c8d3d00014ff0c4d7ed21dd1b03ee1de"
server_key_iv: "88ccade641dde7735c9223926f5f0c6c"
server_key_version: "encryptionkey@0"
I20260925 05:01:22.443516 5873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-1/wal/instance:
uuid: "543dab666dc748caa9542055c3b85e43"
format_stamp: "Formatted at 2026-09-25 05:01:22 on dist-test-slave-5n86"
server_key: "c8d3d00014ff0c4d7ed21dd1b03ee1de"
server_key_iv: "88ccade641dde7735c9223926f5f0c6c"
server_key_version: "encryptionkey@0"
I20260925 05:01:22.448362 5873 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 05:01:22.451532 5888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:22.452924 5873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:01:22.453090 5873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-1/wal
uuid: "543dab666dc748caa9542055c3b85e43"
format_stamp: "Formatted at 2026-09-25 05:01:22 on dist-test-slave-5n86"
server_key: "c8d3d00014ff0c4d7ed21dd1b03ee1de"
server_key_iv: "88ccade641dde7735c9223926f5f0c6c"
server_key_version: "encryptionkey@0"
I20260925 05:01:22.453297 5873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:22.468784 5873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:22.472945 5873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:22.473193 5873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:22.491009 5873 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.61:42081
I20260925 05:01:22.491029 5939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.61:42081 every 8 connection(s)
I20260925 05:01:22.492290 5873 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-1/data/info.pb
I20260925 05:01:22.495605 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 5873
I20260925 05:01:22.495751 5940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:22.505339 5940 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:01:22 dist-test-slave-5n86 krb5kdc[5367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.247.61@KRBTEST.COM
Sep 25 05:01:22 dist-test-slave-5n86 krb5kdc[5367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312482, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.61@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM
I20260925 05:01:22.519765 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:22.498319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39976 (local address 127.30.247.61:42081)
0925 05:01:22.499112 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:22.499127 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:22.499175 (+ 48us) server_negotiation.cc:407] Connection header received
0925 05:01:22.500794 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:22.500825 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:22.501317 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:22.501755 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:22.503070 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.505019 (+ 1949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.506387 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.507345 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.507495 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:22.511908 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:22.511918 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:22.511930 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:22.511976 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:22.515811 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:22.516608 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:22.516617 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:22.516624 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:22.516801 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:22.517280 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:22.517285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:22.517288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:22.517862 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:22.518152 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:22.518570 (+ 418us) server_negotiation.cc:299] Negotiation successful
0925 05:01:22.518926 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":351,"threads_started":1}
I20260925 05:01:22.524396 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:22.507919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49674 (local address 127.30.247.62:44265)
0925 05:01:22.508086 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:22.508090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:22.508846 (+ 756us) server_negotiation.cc:407] Connection header received
0925 05:01:22.509073 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:22.509079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:22.509149 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:22.509252 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:22.510242 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.511285 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.513173 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.513779 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.513939 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:22.517772 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:22.517778 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:22.517783 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:22.517820 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:22.520687 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:22.522514 (+ 1827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:22.522522 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:22.522525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:22.522651 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:22.523017 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:22.523021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:22.523024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:22.523304 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:22.523450 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:22.523874 (+ 424us) server_negotiation.cc:299] Negotiation successful
0925 05:01:22.524075 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 05:01:22.524470 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:22.507929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.0.0.1:49674)
0925 05:01:22.508717 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:22.508755 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:22.508924 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:22.509307 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:22.509314 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:22.509872 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:22.510103 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:22.510110 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.511415 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.511420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:22.513020 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:22.513042 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.513980 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.513995 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:22.514201 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:22.514807 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:22.514829 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:22.517588 (+ 2759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:22.521863 (+ 4275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:22.521874 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:22.521880 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:22.522381 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:22.522758 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:22.522764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:22.522767 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:22.522902 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:22.523444 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:22.523451 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:22.523610 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 05:01:22.523772 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 05:01:22.524062 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":377,"threads_started":1}
I20260925 05:01:22.526015 5940 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:01:22 dist-test-slave-5n86 krb5kdc[5367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312482, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.61@KRBTEST.COM for oryx/127.30.247.61@KRBTEST.COM
I20260925 05:01:22.537297 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:22.526663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:42081 (local address 127.0.0.1:39990)
0925 05:01:22.526882 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:22.526901 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:22.527000 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:22.527248 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:22.527251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:22.527547 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:22.527668 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:22.527675 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.528642 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.528645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:22.529493 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:22.529503 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.530133 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.530137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:22.530270 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:22.530723 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:22.530740 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:22.532700 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:22.535311 (+ 2611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:22.535314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:22.535317 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:22.535675 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:22.535990 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:22.535993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:22.535995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:22.536108 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:22.536608 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:22.536611 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:22.536724 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 05:01:22.536833 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 05:01:22.537038 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 05:01:22.537377 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:22.526885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39990 (local address 127.30.247.61:42081)
0925 05:01:22.527024 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:22.527028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:22.527042 (+ 14us) server_negotiation.cc:407] Connection header received
0925 05:01:22.527077 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:22.527080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:22.527127 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:22.527198 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:22.527788 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.528459 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.529609 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.529992 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.530083 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:22.532845 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:22.532849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:22.532852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:22.532877 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:22.535185 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:22.535776 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:22.535779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:22.535781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:22.535886 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:22.536192 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:22.536195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:22.536197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:22.536444 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:22.536616 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:22.536907 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 05:01:22.537147 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":53}
I20260925 05:01:22.541631 5940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43: Bootstrap starting.
I20260925 05:01:22.544476 5940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:22.545504 5940 log.cc:824] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43: Log is configured to *not* fsync() on all Append() calls
I20260925 05:01:22.546548 5735 heartbeater.cc:503] Master 127.30.247.62:44265 was elected leader, sending a full tablet report...
I20260925 05:01:22.548272 5940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43: No bootstrap required, opened a new log
I20260925 05:01:22.554955 5940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } }
I20260925 05:01:22.555280 5940 raft_consensus.cc:374] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:22.555387 5940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 543dab666dc748caa9542055c3b85e43, State: Initialized, Role: FOLLOWER
I20260925 05:01:22.555948 5940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } }
I20260925 05:01:22.556970 5946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } } }
I20260925 05:01:22.557299 5946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:01:22.557884 5940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 05:01:22.566874 5957 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:01:22.567016 5957 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:01:22.568642 5873 master_runner.cc:428] Detected that this master 543dab666dc748caa9542055c3b85e43 is joining an existing cluster
I20260925 05:01:22.568745 5873 master_runner.cc:432] Initiating AddMaster RPC to add 127.30.247.61:42081
WARNING: no policy specified for oryx/127.30.247.60@KRBTEST.COM; defaulting to no policy
I20260925 05:01:22.601620 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:22.572278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39992 (local address 127.30.247.61:42081)
0925 05:01:22.572468 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:22.572474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:22.585619 (+ 13145us) server_negotiation.cc:407] Connection header received
0925 05:01:22.585833 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:22.585838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:22.585896 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:22.585968 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:22.587390 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.588500 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.590511 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.591178 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.591271 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:22.593861 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:22.593869 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:22.593879 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:22.593911 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:22.598177 (+ 4266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:22.599308 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:22.599314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:22.599320 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:22.599469 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:22.599923 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:22.599931 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:22.599934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:22.600328 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:22.600456 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:22.601022 (+ 566us) server_negotiation.cc:299] Negotiation successful
0925 05:01:22.601196 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 05:01:22.602407 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:22.572850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:42081 (local address 127.0.0.1:39992)
0925 05:01:22.585523 (+ 12673us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:22.585545 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:22.585705 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:22.586139 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:22.586144 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:22.586922 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:22.587224 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:22.587240 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.588752 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.588757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:22.590370 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:22.590383 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.591567 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.591574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:22.591747 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:22.592468 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:22.592490 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:22.593674 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:22.598323 (+ 4649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:22.598329 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:22.598333 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:22.599172 (+ 839us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:22.599609 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:22.599616 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:22.599623 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:22.599798 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:22.600579 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:22.600584 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:22.600739 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 05:01:22.601886 (+ 1147us) client_negotiation.cc:240] Negotiation successful
0925 05:01:22.602127 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12538,"mutex_wait_us":1,"thread_start_us":12336,"threads_started":1}
I20260925 05:01:22.603600 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:22.572273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49678 (local address 127.30.247.62:44265)
0925 05:01:22.572726 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:22.572735 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:22.584669 (+ 11934us) server_negotiation.cc:407] Connection header received
0925 05:01:22.584868 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:22.584877 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:22.584986 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:22.585125 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:22.586852 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.588159 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.590732 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.591525 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.591636 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:22.597076 (+ 5440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:22.597083 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:22.597088 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:22.597129 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:22.600569 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:22.601310 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:22.601315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:22.601319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:22.601459 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:22.601900 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:22.601906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:22.601909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:22.602249 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:22.602388 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:22.603019 (+ 631us) server_negotiation.cc:299] Negotiation successful
0925 05:01:22.603235 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320}
I20260925 05:01:22.604525 5963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:22.572343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.0.0.1:49678)
0925 05:01:22.584518 (+ 12175us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:22.584565 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:22.584718 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:22.585389 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:22.585394 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:22.586158 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:22.586672 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:22.586692 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.588358 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.588366 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:22.590554 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:22.590571 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.591922 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.591930 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:22.592526 (+ 596us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:22.593224 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:22.593250 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:22.596831 (+ 3581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:22.600719 (+ 3888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:22.600724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:22.600727 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:22.601163 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:22.601575 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:22.601580 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:22.601584 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:22.601766 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:22.602475 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:22.602480 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:22.602698 (+ 218us) client_negotiation.cc:769] Sending connection context
0925 05:01:22.603774 (+ 1076us) client_negotiation.cc:240] Negotiation successful
0925 05:01:22.604082 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12018,"mutex_wait_us":4,"thread_start_us":12468,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.247.60@KRBTEST.COM" created.
Sep 25 05:01:22 dist-test-slave-5n86 krb5kdc[5367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312480, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.62@KRBTEST.COM for oryx/127.30.247.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.247.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.60.keytab.
Entry for principal oryx/127.30.247.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.60.keytab.
I20260925 05:01:22.633379 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.60:41659
--webserver_interface=127.30.247.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.60.keytab
--principal=oryx/127.30.247.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:44265,127.30.247.61:42081,127.30.247.60:41659
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:01:22.633965 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:22.617792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40002 (local address 127.30.247.61:42081)
0925 05:01:22.617961 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:22.617967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:22.618607 (+ 640us) server_negotiation.cc:407] Connection header received
0925 05:01:22.618955 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:22.618967 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:22.619075 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:22.619183 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:22.620508 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.621758 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.623664 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.624444 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.624625 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:22.627923 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:22.627928 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:22.627931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:22.627959 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:22.630785 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:22.631541 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:22.631546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:22.631550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:22.631713 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:22.632244 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:22.632250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:22.632253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:22.632665 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:22.632817 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:22.633400 (+ 583us) server_negotiation.cc:299] Negotiation successful
0925 05:01:22.633610 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 05:01:22.634469 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:22.617801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:42081 (local address 127.0.0.1:40002)
0925 05:01:22.618480 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:22.618522 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:22.618738 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:22.619250 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:22.619258 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:22.620040 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:22.620311 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:22.620323 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.621907 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.621914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:22.623459 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:22.623480 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.624661 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.624670 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:22.624871 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:22.625551 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:22.625568 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:22.627742 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:22.630915 (+ 3173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:22.630928 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:22.630935 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:22.631413 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:22.631917 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:22.631922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:22.631926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:22.632115 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:22.632807 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:22.632819 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:22.633063 (+ 244us) client_negotiation.cc:769] Sending connection context
0925 05:01:22.633291 (+ 228us) client_negotiation.cc:240] Negotiation successful
0925 05:01:22.633791 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":285,"threads_started":1}
I20260925 05:01:22.636464 5412 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.30.247.61:42081: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "543dab666dc748caa9542055c3b85e43"
member_type: NON_VOTER
last_known_addr {
host: "127.30.247.61"
port: 42081
}
attrs {
promote: true
}
}
dest_uuid: "190a829b060c49298dcf7111c5dab8c6"
cas_config_opid_index: -1
I20260925 05:01:22.637631 5412 consensus_queue.cc:237] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true } }
I20260925 05:01:22.645061 5915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:22.644496 5971 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 543dab666dc748caa9542055c3b85e43 (127.30.247.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true } } }
I20260925 05:01:22.648789 5915 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [term 1 FOLLOWER]: Refusing update from remote peer 190a829b060c49298dcf7111c5dab8c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 05:01:22.649353 5971 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.30.247.61:42081
I20260925 05:01:22.650041 5873 master.cc:537] Master@127.30.247.61:42081 shutting down...
I20260925 05:01:22.659157 5970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "190a829b060c49298dcf7111c5dab8c6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true } } }
I20260925 05:01:22.659657 5970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:22.660147 5971 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:01:22.663180 5873 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:01:22.663570 5873 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:01:22.663743 5873 tablet_replica.cc:330] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43: stopping tablet replica
W20260925 05:01:22.665448 5399 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260925 05:01:22.669996 5873 master.cc:552] Master@127.30.247.61:42081 shutdown complete.
I20260925 05:01:22.670075 5873 master_runner.cc:305] Clearing existing system tablet
I20260925 05:01:22.670523 5873 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:22.670616 5873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260925 05:01:22.671713 5399 consensus_peers.cc:487] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 -> Peer 543dab666dc748caa9542055c3b85e43 (127.30.247.61:42081): Couldn't send request to peer 543dab666dc748caa9542055c3b85e43. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260925 05:01:22.673447 5873 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260925 05:01:22.675031 5976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:22.675365 5873 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:22.675760 5873 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260925 05:01:22.675901 5873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-1/wal
uuid: "543dab666dc748caa9542055c3b85e43"
format_stamp: "Formatted at 2026-09-25 05:01:22 on dist-test-slave-5n86"
server_key: "c8d3d00014ff0c4d7ed21dd1b03ee1de"
server_key_iv: "88ccade641dde7735c9223926f5f0c6c"
server_key_version: "encryptionkey@0"
I20260925 05:01:22.680724 5873 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:01:22.687495 5873 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 05:01:22.687713 5873 log.cc:1196] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 05:01:22.688175 5873 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43: Deleting consensus metadata
I20260925 05:01:22.688493 5873 master_runner.cc:315] Copying system tablet from 127.30.247.62:44265
I20260925 05:01:22.690981 5873 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43: tablet copy: Beginning tablet copy session from remote peer at address 127.30.247.62:44265
I20260925 05:01:22.706615 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:22.691885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49692 (local address 127.30.247.62:44265)
0925 05:01:22.692059 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:22.692065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:22.692494 (+ 429us) server_negotiation.cc:407] Connection header received
0925 05:01:22.692845 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:22.692852 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:22.692941 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:22.693012 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:22.694152 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.695262 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.697104 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.697732 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.697824 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:22.700210 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:22.700217 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:22.700222 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:22.700258 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:22.703578 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:22.704399 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:22.704404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:22.704408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:22.704582 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:22.705007 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:22.705012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:22.705016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:22.705369 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:22.705484 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:22.706024 (+ 540us) server_negotiation.cc:299] Negotiation successful
0925 05:01:22.706247 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:01:22.707732 5982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:22.691763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.0.0.1:49692)
0925 05:01:22.692393 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:22.692416 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:22.692691 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:22.693205 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:22.693210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:22.693723 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:22.693978 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:22.693991 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.695462 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.695468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:22.696945 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:22.696964 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:22.698062 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:22.698069 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:22.698247 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:22.698994 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:22.699015 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:22.700002 (+ 987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:22.703724 (+ 3722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:22.703729 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:22.703734 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:22.704255 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:22.704725 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:22.704730 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:22.704734 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:22.704886 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:22.705577 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:22.705587 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:22.705751 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 05:01:22.707018 (+ 1267us) client_negotiation.cc:240] Negotiation successful
0925 05:01:22.707298 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":397,"threads_started":1}
I20260925 05:01:22.708796 5432 tablet_copy_service.cc:140] P 190a829b060c49298dcf7111c5dab8c6: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 543dab666dc748caa9542055c3b85e43 ({username='oryx', principal='oryx/127.30.247.61@KRBTEST.COM'} at 127.0.0.1:49692)
I20260925 05:01:22.709084 5432 tablet_copy_service.cc:161] P 190a829b060c49298dcf7111c5dab8c6: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 543dab666dc748caa9542055c3b85e43 at {username='oryx', principal='oryx/127.30.247.61@KRBTEST.COM'} at 127.0.0.1:49692: session id = 543dab666dc748caa9542055c3b85e43-00000000000000000000000000000000
I20260925 05:01:22.713486 5432 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 05:01:22.715505 5873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 05:01:22.722168 5873 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43: tablet copy: Starting download of 0 data blocks...
I20260925 05:01:22.722464 5873 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43: tablet copy: Starting download of 1 WAL segments...
I20260925 05:01:22.725111 5873 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 05:01:22.729157 5432 tablet_copy_service.cc:342] P 190a829b060c49298dcf7111c5dab8c6: Request end of tablet copy session 543dab666dc748caa9542055c3b85e43-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.30.247.61@KRBTEST.COM'} at 127.0.0.1:49692
I20260925 05:01:22.729351 5432 tablet_copy_service.cc:433] P 190a829b060c49298dcf7111c5dab8c6: ending tablet copy session 543dab666dc748caa9542055c3b85e43-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 543dab666dc748caa9542055c3b85e43
I20260925 05:01:22.740399 5873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:22.742936 5983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:22.743415 5984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:22.747776 5873 server_base.cc:1034] running on GCE node
W20260925 05:01:22.748121 5986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:22.748622 5873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:22.749315 5873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:22.750571 5873 hybrid_clock.cc:648] HybridClock initialized: now 1790312482750513 us; error 35 us; skew 500 ppm
I20260925 05:01:22.750777 5873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:22 dist-test-slave-5n86 krb5kdc[5367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312482, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:22.752925 5873 init.cc:377] Logged in from keytab as oryx/127.30.247.61@KRBTEST.COM (short username oryx)
I20260925 05:01:22.754091 5873 webserver.cc:460] Webserver started at http://127.30.247.61:39577/ using document root <none> and password file <none>
I20260925 05:01:22.754433 5873 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:22.754513 5873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:22.757413 5873 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 05:01:22.759236 5993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:22.759661 5873 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:01:22.759793 5873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-1/wal
uuid: "543dab666dc748caa9542055c3b85e43"
format_stamp: "Formatted at 2026-09-25 05:01:22 on dist-test-slave-5n86"
server_key: "c8d3d00014ff0c4d7ed21dd1b03ee1de"
server_key_iv: "88ccade641dde7735c9223926f5f0c6c"
server_key_version: "encryptionkey@0"
I20260925 05:01:22.759949 5873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:22.774693 5873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:22.778879 5873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:22.810468 5873 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.61:42081
I20260925 05:01:22.810539 6044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.61:42081 every 8 connection(s)
I20260925 05:01:22.811474 5873 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-1/data/info.pb
I20260925 05:01:22.816814 6045 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:01:22.820806 6045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43: Bootstrap starting.
I20260925 05:01:22.838974 6045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:01:22.839613 6045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43: Bootstrap complete.
I20260925 05:01:22.840943 6045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true } }
I20260925 05:01:22.841288 6045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 543dab666dc748caa9542055c3b85e43, State: Initialized, Role: LEARNER
I20260925 05:01:22.841711 6045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true } }
I20260925 05:01:22.842660 6048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true } } }
I20260925 05:01:22.843062 6045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:01:22.843179 6048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [sys.catalog]: This master's current role is: LEARNER
I20260925 05:01:22.854264 6059 catalog_manager.cc:1259] Loaded cluster ID: 69f5d87d0fd1476db9d81f6d4c7fdfba
I20260925 05:01:22.854372 6059 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43: loading cluster ID for follower catalog manager: success
I20260925 05:01:22.857746 6059 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43: acquiring CA information for follower catalog manager: success
I20260925 05:01:22.860128 6059 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 05:01:23.014374 5968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:23.014688 5968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:23.014839 5968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:23.022001 5968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:01:23.022128 5968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:23.022238 5968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:23.022336 5968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:01:23.022418 5968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:01:23.033461 5968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:44265,127.30.247.61:42081,127.30.247.60:41659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.60.keytab
--principal=oryx/127.30.247.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.60:41659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.247.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:23.034646 5968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:23.035874 5968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:01:23.046120 5868 heartbeater.cc:503] Master 127.30.247.62:44265 was elected leader, sending a full tablet report...
W20260925 05:01:23.048601 6064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:23.048554 6065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:23.048835 6067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:23.049746 5968 server_base.cc:1034] running on GCE node
I20260925 05:01:23.050437 5968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:23.051729 5968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:23.052978 5968 hybrid_clock.cc:648] HybridClock initialized: now 1790312483052918 us; error 49 us; skew 500 ppm
I20260925 05:01:23.053325 5968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:23 dist-test-slave-5n86 krb5kdc[5367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312483, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:23.056408 5968 init.cc:377] Logged in from keytab as oryx/127.30.247.60@KRBTEST.COM (short username oryx)
I20260925 05:01:23.057946 5968 webserver.cc:460] Webserver started at http://127.30.247.60:38761/ using document root <none> and password file <none>
I20260925 05:01:23.058605 5968 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:23.058745 5968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:23.059036 5968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:23.062166 5968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-2/data/instance:
uuid: "b27f6d41f6f1426bb70abaf6985e00b2"
format_stamp: "Formatted at 2026-09-25 05:01:23 on dist-test-slave-5n86"
server_key: "54e224b315b910c0aa1b8a8cf3fb64ce"
server_key_iv: "cc3fed379496533ce0eb1ffe3538a2c9"
server_key_version: "encryptionkey@0"
I20260925 05:01:23.063035 5968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-2/wal/instance:
uuid: "b27f6d41f6f1426bb70abaf6985e00b2"
format_stamp: "Formatted at 2026-09-25 05:01:23 on dist-test-slave-5n86"
server_key: "54e224b315b910c0aa1b8a8cf3fb64ce"
server_key_iv: "cc3fed379496533ce0eb1ffe3538a2c9"
server_key_version: "encryptionkey@0"
I20260925 05:01:23.068269 5968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 05:01:23.071745 6074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:23.073323 5968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:01:23.073508 5968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-2/wal
uuid: "b27f6d41f6f1426bb70abaf6985e00b2"
format_stamp: "Formatted at 2026-09-25 05:01:23 on dist-test-slave-5n86"
server_key: "54e224b315b910c0aa1b8a8cf3fb64ce"
server_key_iv: "cc3fed379496533ce0eb1ffe3538a2c9"
server_key_version: "encryptionkey@0"
I20260925 05:01:23.073700 5968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:23.101624 5968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:23.105849 5968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:23.106133 5968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:23.123669 5968 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.60:41659
I20260925 05:01:23.123700 6125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.60:41659 every 8 connection(s)
I20260925 05:01:23.125025 5968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-2/data/info.pb
I20260925 05:01:23.127847 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 5968
I20260925 05:01:23.128600 6126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:23.136426 6126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:01:23 dist-test-slave-5n86 krb5kdc[5367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.247.60@KRBTEST.COM
Sep 25 05:01:23 dist-test-slave-5n86 krb5kdc[5367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312483, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.60@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM
I20260925 05:01:23.151878 6129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.129936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54736 (local address 127.30.247.60:41659)
0925 05:01:23.130644 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:23.130655 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:23.130691 (+ 36us) server_negotiation.cc:407] Connection header received
0925 05:01:23.132947 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:23.132972 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:23.133333 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:23.133710 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:23.135132 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.136362 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.137856 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.138754 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.138956 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.143938 (+ 4982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:23.143949 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:23.143961 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:23.144001 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:23.147507 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.148266 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.148274 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.148280 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.148416 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.148836 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.148843 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.148848 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.149512 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:23.149858 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:23.150164 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 05:01:23.150558 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":313,"threads_started":1}
I20260925 05:01:23.155234 6130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.138665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.0.0.1:49702)
0925 05:01:23.139650 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:23.139695 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:23.139831 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:23.140201 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:23.140206 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:23.140818 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:23.141076 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:23.141095 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.142428 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.142436 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:23.144015 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:23.144026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.144921 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.144928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:23.145174 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.145757 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:23.145785 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:23.148489 (+ 2704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:23.152808 (+ 4319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:23.152818 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:23.152824 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:23.153406 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:23.153762 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:23.153766 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:23.153769 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:23.153914 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:23.154443 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:23.154450 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:23.154639 (+ 189us) client_negotiation.cc:769] Sending connection context
0925 05:01:23.154786 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 05:01:23.154989 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":652,"threads_started":1}
I20260925 05:01:23.155239 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.138884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49702 (local address 127.30.247.62:44265)
0925 05:01:23.139110 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:23.139119 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:23.139790 (+ 671us) server_negotiation.cc:407] Connection header received
0925 05:01:23.139961 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:23.139967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:23.140054 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:23.140159 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:23.141231 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.142284 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.144158 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.144730 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.144853 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.148663 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:23.148668 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:23.148671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:23.148720 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:23.152682 (+ 3962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.153540 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.153544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.153546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.153657 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.154033 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.154036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.154038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.154311 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:23.154470 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:23.154875 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 05:01:23.155036 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 05:01:23.156509 6126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:01:23 dist-test-slave-5n86 krb5kdc[5367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312483, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.60@KRBTEST.COM for oryx/127.30.247.61@KRBTEST.COM
I20260925 05:01:23.169260 6130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.157154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:42081 (local address 127.0.0.1:40012)
0925 05:01:23.157324 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:23.157341 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:23.157430 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:23.158074 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:23.158078 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:23.158357 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:23.158475 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:23.158482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.159679 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.159682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:23.160723 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:23.160733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.161396 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.161402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:23.161649 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.162205 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:23.162221 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:23.164034 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:23.166990 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:23.166998 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:23.167001 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:23.167352 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:23.167728 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:23.167734 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:23.167738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:23.167864 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:23.168347 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:23.168353 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:23.168493 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 05:01:23.168657 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 05:01:23.168939 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 05:01:23.169317 6131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.157297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40012 (local address 127.30.247.61:42081)
0925 05:01:23.157795 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:23.157799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:23.157813 (+ 14us) server_negotiation.cc:407] Connection header received
0925 05:01:23.157859 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:23.157863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:23.157915 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:23.158006 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:23.158620 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.159574 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.160835 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.161233 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.161386 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.164211 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:23.164215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:23.164219 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:23.164247 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:23.166857 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.167479 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.167487 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.167490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.167620 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.167983 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.167990 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.167993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.168233 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:23.168393 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:23.168762 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 05:01:23.168973 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":288,"threads_started":1}
I20260925 05:01:23.170313 6126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.60" port: 41659 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:01:23 dist-test-slave-5n86 krb5kdc[5367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312483, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.60@KRBTEST.COM for oryx/127.30.247.60@KRBTEST.COM
I20260925 05:01:23.182587 6129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.171081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54742 (local address 127.30.247.60:41659)
0925 05:01:23.171271 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:23.171277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:23.171363 (+ 86us) server_negotiation.cc:407] Connection header received
0925 05:01:23.171521 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:23.171525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:23.171585 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:23.171676 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:23.172268 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.172994 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.174027 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.174409 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.174546 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.177535 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:23.177561 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:23.177568 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:23.177602 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:23.180214 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.180849 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.180852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.180855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.180980 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.181325 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.181328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.181330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.181613 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:23.181713 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:23.182048 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 05:01:23.182276 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":74}
I20260925 05:01:23.182615 6130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.171101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:41659 (local address 127.0.0.1:54742)
0925 05:01:23.171275 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:23.171292 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:23.171417 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:23.171688 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:23.171692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:23.172032 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:23.172147 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:23.172154 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.173107 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.173114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:23.173900 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:23.173910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.174557 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.174562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:23.174760 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.175275 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:23.175297 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:23.177380 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:23.180344 (+ 2964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:23.180347 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:23.180350 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:23.180757 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:23.181081 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:23.181085 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:23.181087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:23.181237 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:23.181710 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:23.181713 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:23.181858 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 05:01:23.181975 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 05:01:23.182146 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":1}
I20260925 05:01:23.187407 6126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2: Bootstrap starting.
I20260925 05:01:23.190724 6126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:23.192076 6126 log.cc:824] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2: Log is configured to *not* fsync() on all Append() calls
I20260925 05:01:23.195182 6126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2: No bootstrap required, opened a new log
I20260925 05:01:23.199222 6131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.194025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40024 (local address 127.30.247.61:42081)
0925 05:01:23.194207 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:23.194213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:23.194360 (+ 147us) server_negotiation.cc:407] Connection header received
0925 05:01:23.194567 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:23.194572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:23.194667 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:23.194778 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:01:23.195156 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.195904 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.197494 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.198293 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.198445 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.198638 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:23.198838 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 05:01:23.198976 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 05:01:23.199263 6134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.193715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:42081 (local address 127.0.0.1:40024)
0925 05:01:23.194257 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:23.194274 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:23.194437 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:23.194808 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:23.194813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:23.194838 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:01:23.195026 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:23.195034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.196040 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.196046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:23.197339 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:23.197351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.198484 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.198490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:23.198647 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.198704 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 05:01:23.198816 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 05:01:23.198949 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":343,"threads_started":1}
I20260925 05:01:23.200139 6020 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260925 05:01:23.200640 6048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 190a829b060c49298dcf7111c5dab8c6. Latest consensus state: current_term: 1 leader_uuid: "190a829b060c49298dcf7111c5dab8c6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true } } }
I20260925 05:01:23.200886 6048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [sys.catalog]: This master's current role is: LEARNER
I20260925 05:01:23.202478 6126 raft_consensus.cc:348] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 41659 } }
I20260925 05:01:23.202739 6126 raft_consensus.cc:374] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:23.202826 6126 raft_consensus.cc:729] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b27f6d41f6f1426bb70abaf6985e00b2, State: Initialized, Role: FOLLOWER
I20260925 05:01:23.203279 6126 consensus_queue.cc:260] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 41659 } }
I20260925 05:01:23.204092 6135 sys_catalog.cc:455] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 41659 } } }
I20260925 05:01:23.204450 6135 sys_catalog.cc:458] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:01:23.205384 6126 sys_catalog.cc:564] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 05:01:23.213666 6146 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:01:23.213791 6146 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:01:23.217635 5968 master_runner.cc:428] Detected that this master b27f6d41f6f1426bb70abaf6985e00b2 is joining an existing cluster
I20260925 05:01:23.217733 5968 master_runner.cc:432] Initiating AddMaster RPC to add 127.30.247.60:41659
I20260925 05:01:23.248088 6135 raft_consensus.cc:482] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:01:23.248493 6135 raft_consensus.cc:504] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 41659 } }
I20260925 05:01:23.250248 6135 leader_election.cc:288] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 190a829b060c49298dcf7111c5dab8c6 (127.30.247.62:44265), 543dab666dc748caa9542055c3b85e43 (127.30.247.61:42081)
I20260925 05:01:23.251334 6129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.221591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54750 (local address 127.30.247.60:41659)
0925 05:01:23.221793 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:23.221798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:23.233807 (+ 12009us) server_negotiation.cc:407] Connection header received
0925 05:01:23.234178 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:23.234186 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:23.234261 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:23.234335 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:23.235613 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.236660 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.239709 (+ 3049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.240510 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.240663 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.243294 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:23.243301 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:23.243306 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:23.243343 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:23.247161 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.247865 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.247870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.247874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.248049 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.248657 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.248663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.248666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.249264 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:23.249393 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:23.249891 (+ 498us) server_negotiation.cc:299] Negotiation successful
0925 05:01:23.250145 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":83}
I20260925 05:01:23.251163 5422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "190a829b060c49298dcf7111c5dab8c6" is_pre_election: true
I20260925 05:01:23.252316 6020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "543dab666dc748caa9542055c3b85e43" is_pre_election: true
I20260925 05:01:23.253517 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.221794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49706 (local address 127.30.247.62:44265)
0925 05:01:23.222023 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:23.222031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:23.232835 (+ 10804us) server_negotiation.cc:407] Connection header received
0925 05:01:23.233477 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:23.233483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:23.233571 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:23.233646 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:23.235285 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.236449 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.238368 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.239089 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.239193 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.241843 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:23.241850 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:23.241856 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:23.241895 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:23.246656 (+ 4761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.247525 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.247532 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.247536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.247695 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.251890 (+ 4195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.251895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.251898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.252256 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:23.252377 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:23.252952 (+ 575us) server_negotiation.cc:299] Negotiation successful
0925 05:01:23.253193 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 05:01:23.253824 6131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.221435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40032 (local address 127.30.247.61:42081)
0925 05:01:23.221771 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:23.221777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:23.238763 (+ 16986us) server_negotiation.cc:407] Connection header received
0925 05:01:23.238998 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:23.239004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:23.239080 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:23.239152 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:23.240673 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.241726 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.243968 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.244719 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.244817 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.247405 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:23.247414 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:23.247421 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:23.247460 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:23.250810 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.251609 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.251614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.251617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.251755 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.252191 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.252196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.252199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.252574 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:23.252689 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:23.253285 (+ 596us) server_negotiation.cc:299] Negotiation successful
0925 05:01:23.253495 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20260925 05:01:23.253868 6078 leader_election.cc:302] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b27f6d41f6f1426bb70abaf6985e00b2; no voters: 190a829b060c49298dcf7111c5dab8c6, 543dab666dc748caa9542055c3b85e43
I20260925 05:01:23.254506 6153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.221394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.0.0.1:49706)
0925 05:01:23.232697 (+ 11303us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:23.232730 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:23.233275 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:23.233850 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:23.233855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:23.234705 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:23.235104 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:23.235116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.236641 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.236646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:23.238212 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:23.238224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.239416 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.239440 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:23.239657 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.240400 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:23.240441 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:23.241636 (+ 1195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:23.246825 (+ 5189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:23.246831 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:23.246839 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:23.247386 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:23.251588 (+ 4202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:23.251592 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:23.251595 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:23.251764 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:23.252465 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:23.252469 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:23.252680 (+ 211us) client_negotiation.cc:769] Sending connection context
0925 05:01:23.253733 (+ 1053us) client_negotiation.cc:240] Negotiation successful
0925 05:01:23.254032 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11092,"mutex_wait_us":51,"thread_start_us":12411,"threads_started":1}
I20260925 05:01:23.254945 6135 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 05:01:23.255241 6154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.222088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:42081 (local address 127.0.0.1:40032)
0925 05:01:23.238666 (+ 16578us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:23.238690 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:23.238848 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:23.239351 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:23.239356 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:23.240023 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:23.240467 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:23.240481 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.241884 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.241889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:23.243781 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:23.243800 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.245133 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.245140 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:23.245328 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.245952 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:23.245978 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:23.247209 (+ 1231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:23.250983 (+ 3774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:23.250987 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:23.250990 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:23.251469 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:23.251899 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:23.251903 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:23.251907 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:23.252058 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:23.252778 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:23.252785 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:23.253002 (+ 217us) client_negotiation.cc:769] Sending connection context
0925 05:01:23.253977 (+ 975us) client_negotiation.cc:240] Negotiation successful
0925 05:01:23.254234 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16463,"mutex_wait_us":3,"thread_start_us":15502,"threads_started":1}
I20260925 05:01:23.255842 6152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.221471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:41659 (local address 127.0.0.1:54750)
0925 05:01:23.233731 (+ 12260us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:23.233748 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:23.234035 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:23.234521 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:23.234525 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:23.235183 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:23.235487 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:23.235497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.236873 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.236878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:23.239518 (+ 2640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:23.239538 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.240926 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.240932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:23.241181 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.241855 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:23.241895 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:23.243092 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:23.247306 (+ 4214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:23.247311 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:23.247314 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:23.247746 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:23.248189 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:23.248193 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:23.248196 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:23.248504 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:23.249469 (+ 965us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:23.249474 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:23.249623 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 05:01:23.250273 (+ 650us) client_negotiation.cc:240] Negotiation successful
0925 05:01:23.250620 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12130,"mutex_wait_us":190,"thread_start_us":11393,"threads_started":1}
I20260925 05:01:23.257625 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 5473
Sep 25 05:01:23 dist-test-slave-5n86 krb5kdc[5367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312480, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.62@KRBTEST.COM for oryx/127.30.247.60@KRBTEST.COM
Sep 25 05:01:23 dist-test-slave-5n86 krb5kdc[5367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.247.1@KRBTEST.COM
I20260925 05:01:23.274497 6134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.260126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:41659 (local address 127.0.0.1:54760)
0925 05:01:23.260356 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:23.260370 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:23.260492 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:23.260931 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:23.260936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:23.261309 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:23.261435 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:23.261443 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.262571 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.262576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:23.263501 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:23.263512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.264412 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.264419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:23.264615 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.265262 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:23.265286 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:23.268043 (+ 2757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:23.271950 (+ 3907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:23.271955 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:23.271959 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:23.272457 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:23.272926 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:23.272931 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:23.272935 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:23.273074 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:23.273572 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:23.273577 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:23.273749 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 05:01:23.273890 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 05:01:23.274153 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260925 05:01:23.274505 6129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.260119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54760 (local address 127.30.247.60:41659)
0925 05:01:23.260257 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:23.260263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:23.260446 (+ 183us) server_negotiation.cc:407] Connection header received
0925 05:01:23.260662 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:23.260669 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:23.260754 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:23.260863 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:23.261578 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.262438 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.263641 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.264257 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.264376 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.268236 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:23.268242 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:23.268247 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:23.268280 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:23.271797 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.272631 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.272637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.272642 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.272785 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.273183 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.273189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.273192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.273443 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:23.273589 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:23.273985 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 05:01:23.274175 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 05:01:23.276168 5412 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.30.247.60:41659: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2"
member_type: NON_VOTER
last_known_addr {
host: "127.30.247.60"
port: 41659
}
attrs {
promote: true
}
}
dest_uuid: "190a829b060c49298dcf7111c5dab8c6"
cas_config_opid_index: 5
I20260925 05:01:23.277256 5412 consensus_queue.cc:237] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: NON_VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: true } }
I20260925 05:01:23.277928 6155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.258932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51142 (local address 127.30.247.1:33901)
0925 05:01:23.259612 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:23.259622 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:23.259651 (+ 29us) server_negotiation.cc:407] Connection header received
0925 05:01:23.259741 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:23.259750 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:23.260037 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:23.261937 (+ 1900us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:23.262180 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.263725 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.265462 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.266170 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.266307 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.271163 (+ 4856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:23.271173 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:23.271186 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:23.271215 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:23.274621 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.275466 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.275475 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.275478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.275625 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.276103 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.276108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.276112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.276594 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:23.276907 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:23.277264 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 05:01:23.277528 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":336,"threads_started":1}
I20260925 05:01:23.279340 5596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:23.279247 6132 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b27f6d41f6f1426bb70abaf6985e00b2 (127.30.247.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: NON_VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: true } } }
I20260925 05:01:23.280056 6020 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [term 1 LEARNER]: Refusing update from remote peer 190a829b060c49298dcf7111c5dab8c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 05:01:23.281749 6101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:23.284762 6132 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.30.247.60:41659
I20260925 05:01:23.285269 6156 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 05:01:23.286866 6101 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 1 FOLLOWER]: Refusing update from remote peer 190a829b060c49298dcf7111c5dab8c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 05:01:23.288151 6132 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: NON_VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:01:23.291795 6156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "190a829b060c49298dcf7111c5dab8c6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: NON_VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: true } } }
I20260925 05:01:23.292119 6156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:23.296137 6020 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b27f6d41f6f1426bb70abaf6985e00b2 (127.30.247.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: NON_VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: true } } }
I20260925 05:01:23.300642 6048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "190a829b060c49298dcf7111c5dab8c6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: NON_VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: true } } }
I20260925 05:01:23.300941 6048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [sys.catalog]: This master's current role is: LEARNER
I20260925 05:01:23.301375 6157 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6: attempting to promote NON_VOTER 543dab666dc748caa9542055c3b85e43 to VOTER
I20260925 05:01:23.302147 6157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: NON_VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: true } }
I20260925 05:01:23.304332 6100 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 1 FOLLOWER]: Refusing update from remote peer 190a829b060c49298dcf7111c5dab8c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 05:01:23.305053 6020 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [term 1 LEARNER]: Refusing update from remote peer 190a829b060c49298dcf7111c5dab8c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 05:01:23.305030 6132 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: NON_VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:01:23.305644 6157 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 05:01:23.308388 6132 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 543dab666dc748caa9542055c3b85e43 (127.30.247.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: NON_VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: true } } }
I20260925 05:01:23.313012 5968 master.cc:537] Master@127.30.247.60:41659 shutting down...
I20260925 05:01:23.314184 6157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "190a829b060c49298dcf7111c5dab8c6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: NON_VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: true } } }
I20260925 05:01:23.314002 6019 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 543dab666dc748caa9542055c3b85e43 (127.30.247.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: NON_VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: true } } }
I20260925 05:01:23.314450 6157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:23.314806 6157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "190a829b060c49298dcf7111c5dab8c6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: NON_VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: true } } }
I20260925 05:01:23.315084 6157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:23.316870 6048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "190a829b060c49298dcf7111c5dab8c6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: NON_VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: true } } }
I20260925 05:01:23.317121 6048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:01:23.324641 6101 raft_consensus.cc:858] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 41659 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: NON_VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: true } }
W20260925 05:01:23.324843 6101 raft_consensus.cc:1565] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 41659 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: NON_VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: true } }
I20260925 05:01:23.340946 6101 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 543dab666dc748caa9542055c3b85e43 (127.30.247.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true } } }
I20260925 05:01:23.345230 6166 sys_catalog.cc:455] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "190a829b060c49298dcf7111c5dab8c6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true } } }
I20260925 05:01:23.345500 6100 raft_consensus.cc:1467] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20260925 05:01:23.345752 6163 sys_catalog.cc:455] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "190a829b060c49298dcf7111c5dab8c6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true } } }
I20260925 05:01:23.348358 6163 sys_catalog.cc:458] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 05:01:23.346215 6135 sys_catalog.cc:455] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 190a829b060c49298dcf7111c5dab8c6. Latest consensus state: current_term: 1 leader_uuid: "190a829b060c49298dcf7111c5dab8c6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true } } }
I20260925 05:01:23.348555 6135 sys_catalog.cc:458] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 05:01:23.348511 6100 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 1 NON_PARTICIPANT]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.5->[1.6-1.7]
I20260925 05:01:23.348573 6166 sys_catalog.cc:458] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 05:01:23.349217 6100 raft_consensus.cc:858] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: NON_VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: NON_VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: true } }
W20260925 05:01:23.349375 6100 raft_consensus.cc:1565] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 1 LEARNER]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: NON_VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: NON_VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: true } }
I20260925 05:01:23.349838 6100 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b27f6d41f6f1426bb70abaf6985e00b2 (127.30.247.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: NON_VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: true } } }
I20260925 05:01:23.352671 6166 sys_catalog.cc:455] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "190a829b060c49298dcf7111c5dab8c6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: NON_VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: true } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: NON_VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: true } } }
I20260925 05:01:23.353039 6166 sys_catalog.cc:458] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [sys.catalog]: This master's current role is: LEARNER
I20260925 05:01:23.353899 6132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "190a829b060c49298dcf7111c5dab8c6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: NON_VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: true } } }
I20260925 05:01:23.354203 6132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:23.355840 5968 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 1 LEARNER]: Raft consensus shutting down.
I20260925 05:01:23.356436 5968 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 1 LEARNER]: Raft consensus is shut down!
I20260925 05:01:23.356668 5968 tablet_replica.cc:330] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2: stopping tablet replica
W20260925 05:01:23.358830 5397 consensus_peers.cc:487] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 -> Peer b27f6d41f6f1426bb70abaf6985e00b2 (127.30.247.60:41659): Couldn't send request to peer b27f6d41f6f1426bb70abaf6985e00b2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 05:01:23.374356 5968 master.cc:552] Master@127.30.247.60:41659 shutdown complete.
I20260925 05:01:23.374441 5968 master_runner.cc:305] Clearing existing system tablet
I20260925 05:01:23.374804 5968 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:23.374909 5968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:23.378450 5968 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 05:01:23.380904 6169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:23.381278 5968 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:23.381534 5968 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260925 05:01:23.381634 5968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-2/wal
uuid: "b27f6d41f6f1426bb70abaf6985e00b2"
format_stamp: "Formatted at 2026-09-25 05:01:23 on dist-test-slave-5n86"
server_key: "54e224b315b910c0aa1b8a8cf3fb64ce"
server_key_iv: "cc3fed379496533ce0eb1ffe3538a2c9"
server_key_version: "encryptionkey@0"
I20260925 05:01:23.386081 5968 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:01:23.392645 5968 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 05:01:23.392789 5968 log.cc:1196] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 05:01:23.393321 5968 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2: Deleting consensus metadata
I20260925 05:01:23.393554 5968 master_runner.cc:315] Copying system tablet from 127.30.247.62:44265
I20260925 05:01:23.401827 5968 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2: tablet copy: Beginning tablet copy session from remote peer at address 127.30.247.62:44265
I20260925 05:01:23.417446 6175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.402611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.0.0.1:49716)
0925 05:01:23.403279 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:23.403299 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:23.403450 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:23.403851 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:23.403856 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:23.404338 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:23.404617 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:23.404629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.406099 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.406104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:23.407714 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:23.407727 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.408738 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.408744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:23.408923 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.409624 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:23.409650 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:23.410696 (+ 1046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:23.414519 (+ 3823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:23.414525 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:23.414529 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:23.414997 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:23.415391 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:23.415395 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:23.415399 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:23.415573 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:23.416151 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:23.416158 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:23.416344 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 05:01:23.416504 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 05:01:23.416862 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":401,"threads_started":1}
I20260925 05:01:23.418692 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.402648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49716 (local address 127.30.247.62:44265)
0925 05:01:23.402889 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:23.402896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:23.403410 (+ 514us) server_negotiation.cc:407] Connection header received
0925 05:01:23.403585 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:23.403591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:23.403674 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:23.403785 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:23.404771 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.405942 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.407878 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.408589 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.408725 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.410943 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:23.410949 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:23.410955 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:23.410995 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:23.414368 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.415115 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.415121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.415125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.415269 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.415688 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.415693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.415696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.416024 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:23.416179 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:23.417858 (+ 1679us) server_negotiation.cc:299] Negotiation successful
0925 05:01:23.418209 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260925 05:01:23.419338 5432 tablet_copy_service.cc:140] P 190a829b060c49298dcf7111c5dab8c6: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b27f6d41f6f1426bb70abaf6985e00b2 ({username='oryx', principal='oryx/127.30.247.60@KRBTEST.COM'} at 127.0.0.1:49716)
I20260925 05:01:23.419509 5432 tablet_copy_service.cc:161] P 190a829b060c49298dcf7111c5dab8c6: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b27f6d41f6f1426bb70abaf6985e00b2 at {username='oryx', principal='oryx/127.30.247.60@KRBTEST.COM'} at 127.0.0.1:49716: session id = b27f6d41f6f1426bb70abaf6985e00b2-00000000000000000000000000000000
I20260925 05:01:23.421989 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 5473
I20260925 05:01:23.423096 5432 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 05:01:23.424911 5968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 05:01:23.432627 5968 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2: tablet copy: Starting download of 0 data blocks...
I20260925 05:01:23.432911 5968 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2: tablet copy: Starting download of 1 WAL segments...
I20260925 05:01:23.435185 5968 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 05:01:23.439409 5432 tablet_copy_service.cc:342] P 190a829b060c49298dcf7111c5dab8c6: Request end of tablet copy session b27f6d41f6f1426bb70abaf6985e00b2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.30.247.60@KRBTEST.COM'} at 127.0.0.1:49716
I20260925 05:01:23.439551 5432 tablet_copy_service.cc:433] P 190a829b060c49298dcf7111c5dab8c6: ending tablet copy session b27f6d41f6f1426bb70abaf6985e00b2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b27f6d41f6f1426bb70abaf6985e00b2
I20260925 05:01:23.444306 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:33901
--local_ip_for_outbound_sockets=127.30.247.1
--tserver_master_addrs=127.30.247.62:44265
--webserver_port=39157
--webserver_interface=127.30.247.1
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.1.keytab
--principal=oryx/127.30.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.30.247.62:44265,127.30.247.61:42081
--tserver_master_addrs=127.30.247.62:44265,127.30.247.61:42081,127.30.247.60:41659 with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:01:23.454916 5968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:23.463853 6177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:23.468122 6178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:23.475039 6180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:23.475940 5968 server_base.cc:1034] running on GCE node
I20260925 05:01:23.476263 5968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:23.476902 5968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:23.478094 5968 hybrid_clock.cc:648] HybridClock initialized: now 1790312483478036 us; error 49 us; skew 500 ppm
I20260925 05:01:23.478315 5968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:23 dist-test-slave-5n86 krb5kdc[5367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312483, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:23.480454 5968 init.cc:377] Logged in from keytab as oryx/127.30.247.60@KRBTEST.COM (short username oryx)
I20260925 05:01:23.484202 5968 webserver.cc:460] Webserver started at http://127.30.247.60:36479/ using document root <none> and password file <none>
I20260925 05:01:23.484584 5968 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:23.484678 5968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:23.487701 5968 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260925 05:01:23.489607 6187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:23.490041 5968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:01:23.490193 5968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-2/wal
uuid: "b27f6d41f6f1426bb70abaf6985e00b2"
format_stamp: "Formatted at 2026-09-25 05:01:23 on dist-test-slave-5n86"
server_key: "54e224b315b910c0aa1b8a8cf3fb64ce"
server_key_iv: "cc3fed379496533ce0eb1ffe3538a2c9"
server_key_version: "encryptionkey@0"
I20260925 05:01:23.490429 5968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:23.507377 5968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:23.511276 5968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:23.534055 5968 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.60:41659
I20260925 05:01:23.534117 6238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.60:41659 every 8 connection(s)
I20260925 05:01:23.535228 5968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/master-2/data/info.pb
I20260925 05:01:23.540767 6239 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:01:23.544611 6239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2: Bootstrap starting.
I20260925 05:01:23.559746 6239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:01:23.560242 6239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2: Bootstrap complete.
I20260925 05:01:23.561326 6239 raft_consensus.cc:348] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: NON_VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: true } }
I20260925 05:01:23.561693 6239 raft_consensus.cc:729] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b27f6d41f6f1426bb70abaf6985e00b2, State: Initialized, Role: LEARNER
I20260925 05:01:23.561959 6239 consensus_queue.cc:260] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: NON_VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: true } }
I20260925 05:01:23.563277 6239 sys_catalog.cc:564] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:01:23.563084 6242 sys_catalog.cc:455] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: NON_VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: true } } }
I20260925 05:01:23.563598 6242 sys_catalog.cc:458] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [sys.catalog]: This master's current role is: LEARNER
I20260925 05:01:23.572757 6253 catalog_manager.cc:1259] Loaded cluster ID: 69f5d87d0fd1476db9d81f6d4c7fdfba
I20260925 05:01:23.572853 6253 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2: loading cluster ID for follower catalog manager: success
I20260925 05:01:23.576144 6253 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2: acquiring CA information for follower catalog manager: success
I20260925 05:01:23.578826 6253 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:01:23.729619 6157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "190a829b060c49298dcf7111c5dab8c6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: NON_VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: true } } }
I20260925 05:01:23.729974 6157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:23.735605 6134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.730005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:41659 (local address 127.0.0.1:54762)
0925 05:01:23.730270 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:23.730289 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:23.730473 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:23.731055 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:23.731060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:23.731101 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:01:23.731282 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:23.731291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.732519 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.732525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:23.733719 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:23.733730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.734820 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.734826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:23.735019 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.735056 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 05:01:23.735156 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 05:01:23.735293 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260925 05:01:23.735625 6254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.730160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54762 (local address 127.30.247.60:41659)
0925 05:01:23.730750 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:23.730755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:23.730769 (+ 14us) server_negotiation.cc:407] Connection header received
0925 05:01:23.730824 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:23.730828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:23.730900 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:23.731030 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:01:23.731442 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.732372 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.733867 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.734648 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.734801 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.735047 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:23.735189 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 05:01:23.735313 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":341,"threads_started":1}
I20260925 05:01:23.736465 6214 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 05:01:23.736862 6242 sys_catalog.cc:455] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 190a829b060c49298dcf7111c5dab8c6. Latest consensus state: current_term: 1 leader_uuid: "190a829b060c49298dcf7111c5dab8c6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: NON_VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: true } } }
I20260925 05:01:23.737152 6242 sys_catalog.cc:458] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [sys.catalog]: This master's current role is: LEARNER
I20260925 05:01:23.737546 6157 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6: attempting to promote NON_VOTER b27f6d41f6f1426bb70abaf6985e00b2 to VOTER
I20260925 05:01:23.738097 6157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: false } }
I20260925 05:01:23.739872 6214 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 1 LEARNER]: Refusing update from remote peer 190a829b060c49298dcf7111c5dab8c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 05:01:23.739835 6019 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [term 1 FOLLOWER]: Refusing update from remote peer 190a829b060c49298dcf7111c5dab8c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 05:01:23.740450 6157 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 05:01:23.740746 6132 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 05:01:23.743316 6132 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b27f6d41f6f1426bb70abaf6985e00b2 (127.30.247.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: false } } }
I20260925 05:01:23.745792 6214 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b27f6d41f6f1426bb70abaf6985e00b2 (127.30.247.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: false } } }
I20260925 05:01:23.747443 6019 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b27f6d41f6f1426bb70abaf6985e00b2 (127.30.247.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: false } } }
I20260925 05:01:23.751225 6048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "190a829b060c49298dcf7111c5dab8c6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: false } } }
I20260925 05:01:23.751498 6048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:01:23.751452 6242 sys_catalog.cc:455] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "190a829b060c49298dcf7111c5dab8c6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: false } } }
I20260925 05:01:23.751720 6242 sys_catalog.cc:458] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:01:23.752034 6156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "190a829b060c49298dcf7111c5dab8c6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: false } } }
I20260925 05:01:23.752311 6156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:23.752285 6157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "190a829b060c49298dcf7111c5dab8c6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: false } } }
I20260925 05:01:23.752501 6157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:23.757098 6132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "190a829b060c49298dcf7111c5dab8c6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: false } } }
I20260925 05:01:23.757359 6132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6 [sys.catalog]: This master's current role is: LEADER
W20260925 05:01:23.814236 6176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:23.814523 6176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:23.814692 6176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:23.821648 6176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:23.821779 6176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:23.821950 6176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:01:23.831126 6176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.1.keytab
--principal=oryx/127.30.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:33901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=39157
--tserver_master_addrs=127.30.247.62:44265,127.30.247.61:42081,127.30.247.60:41659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:23.832334 6176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:23.833626 6176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:23.846318 6262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:23.846463 6264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:23.846351 6261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:23.847121 6176 server_base.cc:1034] running on GCE node
I20260925 05:01:23.847777 6176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:23.848753 6176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:23.849977 6176 hybrid_clock.cc:648] HybridClock initialized: now 1790312483849909 us; error 77 us; skew 500 ppm
I20260925 05:01:23.850293 6176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:23 dist-test-slave-5n86 krb5kdc[5367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312483, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:23.853348 6176 init.cc:377] Logged in from keytab as oryx/127.30.247.1@KRBTEST.COM (short username oryx)
I20260925 05:01:23.854785 6176 webserver.cc:460] Webserver started at http://127.30.247.1:39157/ using document root <none> and password file <none>
I20260925 05:01:23.855477 6176 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:23.855636 6176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:23.861340 6176 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.002s
I20260925 05:01:23.864322 6271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:23.865700 6176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 05:01:23.865890 6176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "41fed7f820404f16aa2d506396ef227c"
format_stamp: "Formatted at 2026-09-25 05:01:20 on dist-test-slave-5n86"
server_key: "57b18f70efd299f5c0c3c4cdc2a1fa5b"
server_key_iv: "7df987ca08ecea377f5d67910f9d3087"
server_key_version: "encryptionkey@0"
I20260925 05:01:23.866827 6176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:23.898412 6176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:23.902112 6176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:23.902377 6176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:23.903262 6176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:23.904651 6176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:23.904735 6176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:23.904850 6176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:23.904951 6176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:23.937127 6176 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:33901
I20260925 05:01:23.937216 6383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:33901 every 8 connection(s)
I20260925 05:01:23.938462 6176 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:01:23.943435 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 6176
I20260925 05:01:23.943876 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 5607
Sep 25 05:01:23 dist-test-slave-5n86 krb5kdc[5367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.247.2@KRBTEST.COM
I20260925 05:01:23.977223 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.944982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54668 (local address 127.30.247.2:43559)
0925 05:01:23.949282 (+ 4300us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:23.949290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:23.949320 (+ 30us) server_negotiation.cc:407] Connection header received
0925 05:01:23.949390 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:23.949400 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:23.949664 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:23.949853 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:23.952661 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.954704 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.957312 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.959041 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.963429 (+ 4388us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.969948 (+ 6519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:23.969961 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:23.969978 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:23.970026 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:23.973668 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.974588 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.974596 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.974600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.974745 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.975151 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.975156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.975160 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.975543 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:23.975817 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:23.976746 (+ 929us) server_negotiation.cc:299] Negotiation successful
0925 05:01:23.976952 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4067,"thread_start_us":3951,"threads_started":1}
I20260925 05:01:23.978471 5730 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 25 05:01:23 dist-test-slave-5n86 krb5kdc[5367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312483, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM
Sep 25 05:01:23 dist-test-slave-5n86 krb5kdc[5367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312483, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.60@KRBTEST.COM
Sep 25 05:01:23 dist-test-slave-5n86 krb5kdc[5367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312483, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.61@KRBTEST.COM
I20260925 05:01:23.992341 6254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.941254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:35573 (local address 127.30.247.60:41659)
0925 05:01:23.941534 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:23.941543 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:23.970537 (+ 28994us) server_negotiation.cc:407] Connection header received
0925 05:01:23.971585 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:23.971592 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:23.971676 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:23.971746 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:23.974380 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.975499 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.978546 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.979212 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.979306 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.985569 (+ 6263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:23.985575 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:23.985580 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:23.985621 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:23.989179 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.989928 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.989933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.989936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.990107 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.990551 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.990556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.990559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.990914 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:23.991041 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:23.991814 (+ 773us) server_negotiation.cc:299] Negotiation successful
0925 05:01:23.992034 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 05:01:23.993621 6388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.941162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:40741 (local address 127.30.247.62:44265)
0925 05:01:23.947721 (+ 6559us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:23.947731 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:23.969703 (+ 21972us) server_negotiation.cc:407] Connection header received
0925 05:01:23.971621 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:23.971626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:23.971721 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:23.971796 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:23.974380 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.975501 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.978546 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.979218 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.979306 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.984145 (+ 4839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:23.984150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:23.984154 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:23.984183 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:23.988187 (+ 4004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.989414 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.989420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.989426 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.989578 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.990345 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.990351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.990355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.990725 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:23.990855 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:23.993024 (+ 2169us) server_negotiation.cc:299] Negotiation successful
0925 05:01:23.993227 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6398,"thread_start_us":6290,"threads_started":1}
I20260925 05:01:23.993871 6393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.950318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:41659 (local address 127.30.247.1:35573)
0925 05:01:23.970427 (+ 20109us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:23.970446 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:23.971373 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:23.972297 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:23.972309 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:23.973349 (+ 1040us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:23.974181 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:23.974206 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.975693 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.975698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:23.978387 (+ 2689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:23.978399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.979919 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.979925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:23.980088 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.982526 (+ 2438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:23.982546 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:23.985366 (+ 2820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:23.989324 (+ 3958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:23.989330 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:23.989335 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:23.989797 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:23.990245 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:23.990251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:23.990256 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:23.990425 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:23.991107 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:23.991111 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:23.991364 (+ 253us) client_negotiation.cc:769] Sending connection context
0925 05:01:23.992528 (+ 1164us) client_negotiation.cc:240] Negotiation successful
0925 05:01:23.992891 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19960,"mutex_wait_us":399,"spinlock_wait_cycles":7296,"thread_start_us":18953,"threads_started":1}
I20260925 05:01:23.994019 6394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.949343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.30.247.1:40741)
0925 05:01:23.969569 (+ 20226us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:23.969593 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:23.971291 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:23.972297 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:23.972309 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:23.973349 (+ 1040us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:23.974181 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:23.974206 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.975694 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.975698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:23.978385 (+ 2687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:23.978399 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.979679 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.979685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:23.979982 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.980856 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:23.980892 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:23.983914 (+ 3022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:23.988345 (+ 4431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:23.988353 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:23.988369 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:23.989286 (+ 917us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:23.989714 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:23.989719 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:23.989725 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:23.989973 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:23.990922 (+ 949us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:23.990929 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:23.991364 (+ 435us) client_negotiation.cc:769] Sending connection context
0925 05:01:23.991688 (+ 324us) client_negotiation.cc:240] Negotiation successful
0925 05:01:23.992155 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20076,"mutex_wait_us":481,"spinlock_wait_cycles":54784,"thread_start_us":20691,"threads_started":1}
I20260925 05:01:23.995450 6385 heartbeater.cc:347] Connected to a master server at 127.30.247.62:44265
I20260925 05:01:23.995772 6385 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:23.996727 6384 heartbeater.cc:347] Connected to a master server at 127.30.247.60:41659
I20260925 05:01:23.996855 6384 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:23.996917 6392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.948660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:42081 (local address 127.30.247.1:59031)
0925 05:01:23.968205 (+ 19545us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:23.968257 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:23.971266 (+ 3009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:23.972298 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:23.972310 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:23.973131 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:23.974194 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:23.974209 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.975716 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.975723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:23.979552 (+ 3829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:23.979576 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.981456 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.981467 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:23.981752 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.984208 (+ 2456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:23.984240 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:23.987626 (+ 3386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:23.992661 (+ 5035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:23.992673 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:23.992680 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:23.993388 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:23.993874 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:23.993894 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:23.993901 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:23.994090 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:23.995742 (+ 1652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:23.995749 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:23.995943 (+ 194us) client_negotiation.cc:769] Sending connection context
0925 05:01:23.996130 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 05:01:23.996396 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19025,"mutex_wait_us":1741,"spinlock_wait_cycles":22912,"thread_start_us":18903,"threads_started":1}
I20260925 05:01:23.996950 6389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:23.941185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:59031 (local address 127.30.247.61:42081)
0925 05:01:23.947969 (+ 6784us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:23.947975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:23.968366 (+ 20391us) server_negotiation.cc:407] Connection header received
0925 05:01:23.971590 (+ 3224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:23.971598 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:23.971690 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:23.971778 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:23.974416 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.975530 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.979876 (+ 4346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:23.980755 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:23.980894 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:23.987925 (+ 7031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:23.987934 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:23.987941 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:23.987990 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:23.992411 (+ 4421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.993549 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.993557 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.993563 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.993729 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:23.994212 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:23.994219 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:23.994224 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:23.995516 (+ 1292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:23.995657 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:23.996232 (+ 575us) server_negotiation.cc:299] Negotiation successful
0925 05:01:23.996442 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6638,"thread_start_us":8401,"threads_started":1}
I20260925 05:01:23.997285 6384 heartbeater.cc:511] Master 127.30.247.60:41659 requested a full tablet report, sending...
I20260925 05:01:23.998317 6385 heartbeater.cc:511] Master 127.30.247.62:44265 requested a full tablet report, sending...
I20260925 05:01:23.998690 6386 heartbeater.cc:347] Connected to a master server at 127.30.247.61:42081
I20260925 05:01:23.998831 6386 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:23.999159 6386 heartbeater.cc:511] Master 127.30.247.61:42081 requested a full tablet report, sending...
I20260925 05:01:23.999749 6204 ts_manager.cc:194] Registered new tserver with Master: 41fed7f820404f16aa2d506396ef227c (127.30.247.1:33901)
I20260925 05:01:24.000803 6010 ts_manager.cc:194] Registered new tserver with Master: 41fed7f820404f16aa2d506396ef227c (127.30.247.1:33901)
I20260925 05:01:24.001104 5412 ts_manager.cc:194] Re-registered known tserver with Master: 41fed7f820404f16aa2d506396ef227c (127.30.247.1:33901)
I20260925 05:01:24.001973 5412 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.247.1@KRBTEST.COM'} at 127.30.247.1:40741
I20260925 05:01:24.079341 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 5607
I20260925 05:01:24.096525 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:43559
--local_ip_for_outbound_sockets=127.30.247.2
--tserver_master_addrs=127.30.247.62:44265
--webserver_port=35787
--webserver_interface=127.30.247.2
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.2.keytab
--principal=oryx/127.30.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.30.247.62:44265,127.30.247.61:42081
--tserver_master_addrs=127.30.247.62:44265,127.30.247.61:42081,127.30.247.60:41659 with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:24.415443 6396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:24.415745 6396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:24.415916 6396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:24.422926 6396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:24.423074 6396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:24.423236 6396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:01:24.432960 6396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.2.keytab
--principal=oryx/127.30.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:43559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=35787
--tserver_master_addrs=127.30.247.62:44265,127.30.247.61:42081,127.30.247.60:41659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:24.434172 6396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:24.435361 6396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:24.448576 6404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:24.448665 6402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:24.448643 6401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:24.450254 6396 server_base.cc:1034] running on GCE node
I20260925 05:01:24.450763 6396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:24.451669 6396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:24.452860 6396 hybrid_clock.cc:648] HybridClock initialized: now 1790312484452806 us; error 47 us; skew 500 ppm
I20260925 05:01:24.453183 6396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:24 dist-test-slave-5n86 krb5kdc[5367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312484, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:24.456311 6396 init.cc:377] Logged in from keytab as oryx/127.30.247.2@KRBTEST.COM (short username oryx)
I20260925 05:01:24.457780 6396 webserver.cc:460] Webserver started at http://127.30.247.2:35787/ using document root <none> and password file <none>
I20260925 05:01:24.458455 6396 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:24.458560 6396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:24.463925 6396 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 05:01:24.466586 6411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:24.468159 6396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:01:24.468333 6396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "87f2a171c1c8411e93504e0b8a47b426"
format_stamp: "Formatted at 2026-09-25 05:01:21 on dist-test-slave-5n86"
server_key: "e3ec09d5ddf6ec3391403317a4562254"
server_key_iv: "ff31bc61fa87c9da58cd4bc48a877b81"
server_key_version: "encryptionkey@0"
I20260925 05:01:24.469363 6396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:24.487463 6396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:24.491398 6396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:24.491679 6396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:24.492628 6396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:24.493896 6396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:24.494001 6396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:24.494108 6396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:24.494213 6396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:24.526823 6396 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:43559
I20260925 05:01:24.526867 6523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:43559 every 8 connection(s)
I20260925 05:01:24.528219 6396 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:01:24.535400 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 6396
I20260925 05:01:24.535749 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 5740
Sep 25 05:01:24 dist-test-slave-5n86 krb5kdc[5367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312484, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM
Sep 25 05:01:24 dist-test-slave-5n86 krb5kdc[5367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312484, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.61@KRBTEST.COM
Sep 25 05:01:24 dist-test-slave-5n86 krb5kdc[5367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312484, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.60@KRBTEST.COM
I20260925 05:01:24.561172 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:24.531720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:38945 (local address 127.30.247.62:44265)
0925 05:01:24.532584 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:24.532590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:24.535608 (+ 3018us) server_negotiation.cc:407] Connection header received
0925 05:01:24.536202 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:24.536209 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:24.536289 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:24.536372 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:24.538525 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:24.539564 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:24.542843 (+ 3279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:24.543514 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:24.543614 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:24.553384 (+ 9770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:24.553391 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:24.553396 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:24.553434 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:24.557860 (+ 4426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:24.558905 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:24.558910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:24.558914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:24.559074 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:24.559483 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:24.559487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:24.559491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:24.559855 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:24.559978 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:24.560590 (+ 612us) server_negotiation.cc:299] Negotiation successful
0925 05:01:24.560801 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":618,"threads_started":1}
I20260925 05:01:24.561332 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:24.531252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:44355 (local address 127.30.247.61:42081)
0925 05:01:24.532232 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:24.532238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:24.541000 (+ 8762us) server_negotiation.cc:407] Connection header received
0925 05:01:24.542587 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:24.542596 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:24.542688 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:24.542798 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:24.544363 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:24.545706 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:24.548165 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:24.548910 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:24.549048 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:24.553731 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:24.553737 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:24.553743 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:24.553784 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:24.557336 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:24.558397 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:24.558404 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:24.558408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:24.558538 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:24.559054 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:24.559060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:24.559062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:24.559413 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:24.559522 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:24.560588 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0925 05:01:24.560833 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":854,"thread_start_us":722,"threads_started":1}
Sep 25 05:01:24 dist-test-slave-5n86 krb5kdc[5367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.247.3@KRBTEST.COM
I20260925 05:01:24.563302 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:24.531395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:48727 (local address 127.30.247.60:41659)
0925 05:01:24.531880 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:24.531886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:24.533710 (+ 1824us) server_negotiation.cc:407] Connection header received
0925 05:01:24.541093 (+ 7383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:24.541102 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:24.541205 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:24.541291 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:24.544643 (+ 3352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:24.545807 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:24.549930 (+ 4123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:24.550598 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:24.550696 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:24.556293 (+ 5597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:24.556303 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:24.556308 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:24.556350 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:24.559869 (+ 3519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:24.560856 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:24.560864 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:24.560871 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:24.561014 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:24.561621 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:24.561626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:24.561629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:24.561982 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:24.562136 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:24.562672 (+ 536us) server_negotiation.cc:299] Negotiation successful
0925 05:01:24.562904 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":444,"threads_started":1}
I20260925 05:01:24.563565 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:24.531943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.30.247.2:38945)
0925 05:01:24.533170 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:24.533235 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:24.535915 (+ 2680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:24.536861 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:24.536872 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:24.537586 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:24.538339 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:24.538361 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:24.539755 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:24.539760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:24.542365 (+ 2605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:24.542380 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:24.543985 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:24.543993 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:24.544270 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:24.548593 (+ 4323us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:24.548632 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:24.553137 (+ 4505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:24.558012 (+ 4875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:24.558018 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:24.558021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:24.558780 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:24.559213 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:24.559218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:24.559221 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:24.559360 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:24.560054 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:24.560059 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:24.560200 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 05:01:24.561458 (+ 1258us) client_negotiation.cc:240] Negotiation successful
0925 05:01:24.562097 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"mutex_wait_us":28,"spinlock_wait_cycles":30976,"thread_start_us":8594,"threads_started":1}
I20260925 05:01:24.563585 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:24.532692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:42081 (local address 127.30.247.2:44355)
0925 05:01:24.540907 (+ 8215us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:24.540925 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:24.542416 (+ 1491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:24.543090 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:24.543097 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:24.543700 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:24.544181 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:24.544197 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:24.545915 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:24.545923 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:24.547963 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:24.547985 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:24.549394 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:24.549399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:24.549573 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:24.550941 (+ 1368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:24.550961 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:24.553527 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:24.557484 (+ 3957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:24.557494 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:24.557511 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:24.558135 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:24.558681 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:24.558686 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:24.558690 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:24.558906 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:24.559674 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:24.559682 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:24.560140 (+ 458us) client_negotiation.cc:769] Sending connection context
0925 05:01:24.561575 (+ 1435us) client_negotiation.cc:240] Negotiation successful
0925 05:01:24.562192 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8086,"mutex_wait_us":262,"spinlock_wait_cycles":14208,"thread_start_us":368,"threads_started":1}
I20260925 05:01:24.564949 6533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:24.532296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:41659 (local address 127.30.247.2:48727)
0925 05:01:24.533356 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:24.533613 (+ 257us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:24.540903 (+ 7290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:24.541557 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:24.541564 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:24.542289 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:24.544432 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:24.544445 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:24.545987 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:24.545992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:24.548106 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:24.548119 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:24.551930 (+ 3811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:24.551936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:24.552147 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:24.553405 (+ 1258us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:24.553425 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:24.556094 (+ 2669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:24.560037 (+ 3943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:24.560043 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:24.560048 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:24.560696 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:24.561167 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:24.561172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:24.561176 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:24.561416 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:24.562219 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:24.562224 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:24.562394 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 05:01:24.563469 (+ 1075us) client_negotiation.cc:240] Negotiation successful
0925 05:01:24.563821 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"mutex_wait_us":151,"thread_start_us":377,"threads_started":1}
I20260925 05:01:24.565105 6526 heartbeater.cc:347] Connected to a master server at 127.30.247.61:42081
I20260925 05:01:24.565141 6525 heartbeater.cc:347] Connected to a master server at 127.30.247.62:44265
I20260925 05:01:24.565423 6525 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:24.566183 6524 heartbeater.cc:347] Connected to a master server at 127.30.247.60:41659
I20260925 05:01:24.566314 6524 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:24.565428 6526 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:24.566318 6525 heartbeater.cc:511] Master 127.30.247.62:44265 requested a full tablet report, sending...
I20260925 05:01:24.566879 6526 heartbeater.cc:511] Master 127.30.247.61:42081 requested a full tablet report, sending...
I20260925 05:01:24.567286 6524 heartbeater.cc:511] Master 127.30.247.60:41659 requested a full tablet report, sending...
I20260925 05:01:24.568324 6010 ts_manager.cc:194] Registered new tserver with Master: 87f2a171c1c8411e93504e0b8a47b426 (127.30.247.2:43559)
I20260925 05:01:24.568378 6204 ts_manager.cc:194] Registered new tserver with Master: 87f2a171c1c8411e93504e0b8a47b426 (127.30.247.2:43559)
I20260925 05:01:24.568394 5412 ts_manager.cc:194] Re-registered known tserver with Master: 87f2a171c1c8411e93504e0b8a47b426 (127.30.247.2:43559)
I20260925 05:01:24.569370 5412 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.247.2@KRBTEST.COM'} at 127.30.247.2:38945
I20260925 05:01:24.571157 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:24.537188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60732 (local address 127.30.247.3:41837)
0925 05:01:24.549696 (+ 12508us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:24.549706 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:24.549735 (+ 29us) server_negotiation.cc:407] Connection header received
0925 05:01:24.549806 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:24.549815 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:24.550093 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:24.550271 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:24.551846 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:24.553343 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:24.555280 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:24.556141 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:24.556270 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:24.562771 (+ 6501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:24.562783 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:24.562800 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:24.562841 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:24.566799 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:24.567720 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:24.567730 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:24.567734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:24.567888 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:24.568316 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:24.568322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:24.568324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:24.568736 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:24.570361 (+ 1625us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:24.570513 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 05:01:24.570761 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12337,"thread_start_us":12169,"threads_started":1}
I20260925 05:01:24.572404 5863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:24.653640 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 5740
I20260925 05:01:24.669685 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:41837
--local_ip_for_outbound_sockets=127.30.247.3
--tserver_master_addrs=127.30.247.62:44265
--webserver_port=34051
--webserver_interface=127.30.247.3
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.3.keytab
--principal=oryx/127.30.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.30.247.62:44265,127.30.247.61:42081
--tserver_master_addrs=127.30.247.62:44265,127.30.247.61:42081,127.30.247.60:41659 with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:24.987166 6537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:24.987418 6537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:24.987596 6537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:24.994843 6537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:24.994962 6537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:24.995141 6537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:01:25.004637 6385 heartbeater.cc:503] Master 127.30.247.62:44265 was elected leader, sending a full tablet report...
I20260925 05:01:25.004982 6537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.3.keytab
--principal=oryx/127.30.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:41837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=34051
--tserver_master_addrs=127.30.247.62:44265,127.30.247.61:42081,127.30.247.60:41659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:25.006407 6537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:25.007686 6537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:25.020385 6542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:25.020426 6543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:25.020471 6545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:25.021158 6537 server_base.cc:1034] running on GCE node
I20260925 05:01:25.021767 6537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:25.022600 6537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:25.023803 6537 hybrid_clock.cc:648] HybridClock initialized: now 1790312485023766 us; error 51 us; skew 500 ppm
I20260925 05:01:25.024114 6537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:25 dist-test-slave-5n86 krb5kdc[5367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312485, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:25.027232 6537 init.cc:377] Logged in from keytab as oryx/127.30.247.3@KRBTEST.COM (short username oryx)
I20260925 05:01:25.028612 6537 webserver.cc:460] Webserver started at http://127.30.247.3:34051/ using document root <none> and password file <none>
I20260925 05:01:25.029217 6537 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:25.029322 6537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:25.034221 6537 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260925 05:01:25.036769 6552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:25.038097 6537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:01:25.038269 6537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "ad8603b082d240c287b1cf61856b67ce"
format_stamp: "Formatted at 2026-09-25 05:01:21 on dist-test-slave-5n86"
server_key: "eda2172d83462013c8036a341c088fe9"
server_key_iv: "23afb2dd63a196566dfbf41b16b12ca4"
server_key_version: "encryptionkey@0"
I20260925 05:01:25.039188 6537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:25.055771 6537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:25.059746 6537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:25.060010 6537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:25.061002 6537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:25.062379 6537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:25.062472 6537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:25.062543 6537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:25.062598 6537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:25.096086 6537 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:41837
I20260925 05:01:25.096179 6664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:41837 every 8 connection(s)
I20260925 05:01:25.097671 6537 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:01:25.098204 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 6537
Sep 25 05:01:25 dist-test-slave-5n86 krb5kdc[5367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312485, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.61@KRBTEST.COM
Sep 25 05:01:25 dist-test-slave-5n86 krb5kdc[5367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312485, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.60@KRBTEST.COM
Sep 25 05:01:25 dist-test-slave-5n86 krb5kdc[5367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312485, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM
I20260925 05:01:25.133304 6671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:25.101674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:44419 (local address 127.30.247.61:42081)
0925 05:01:25.102257 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:25.102266 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:25.105585 (+ 3319us) server_negotiation.cc:407] Connection header received
0925 05:01:25.108566 (+ 2981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:25.108576 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:25.108690 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:25.108797 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:25.113095 (+ 4298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.114179 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.116808 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.117392 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.117496 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:25.122910 (+ 5414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:25.122917 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:25.122925 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:25.122988 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:25.127265 (+ 4277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:25.129656 (+ 2391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:25.129662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:25.129668 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:25.129835 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:25.130432 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:25.130436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:25.130439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:25.130758 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:25.130917 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:25.132482 (+ 1565us) server_negotiation.cc:299] Negotiation successful
0925 05:01:25.132765 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":297,"threads_started":1}
I20260925 05:01:25.135757 6673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:25.101715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:42081 (local address 127.30.247.3:44419)
0925 05:01:25.105453 (+ 3738us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:25.105485 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:25.108339 (+ 2854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:25.109390 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:25.109407 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:25.110765 (+ 1358us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:25.112875 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:25.112912 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.114363 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.114368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:25.116643 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:25.116657 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.117844 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.117850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:25.118129 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:25.119064 (+ 935us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:25.119103 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:25.122629 (+ 3526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:25.127441 (+ 4812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:25.127449 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:25.127465 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:25.128127 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:25.129992 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:25.129997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:25.130000 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:25.130288 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:25.131012 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:25.131019 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:25.131402 (+ 383us) client_negotiation.cc:769] Sending connection context
0925 05:01:25.133777 (+ 2375us) client_negotiation.cc:240] Negotiation successful
0925 05:01:25.134307 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2793,"mutex_wait_us":83,"spinlock_wait_cycles":71040,"thread_start_us":2582,"threads_started":1}
I20260925 05:01:25.138092 6667 heartbeater.cc:347] Connected to a master server at 127.30.247.61:42081
I20260925 05:01:25.138432 6667 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:25.139287 6667 heartbeater.cc:511] Master 127.30.247.61:42081 requested a full tablet report, sending...
I20260925 05:01:25.141145 6010 ts_manager.cc:194] Registered new tserver with Master: ad8603b082d240c287b1cf61856b67ce (127.30.247.3:41837)
I20260925 05:01:25.141295 6669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:25.100946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:55505 (local address 127.30.247.60:41659)
0925 05:01:25.112841 (+ 11895us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:25.112849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:25.112873 (+ 24us) server_negotiation.cc:407] Connection header received
0925 05:01:25.112927 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:25.112930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:25.112992 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:25.113131 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:25.114156 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.115215 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.117019 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.117635 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.117742 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:25.133395 (+ 15653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:25.133403 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:25.133409 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:25.133453 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:25.136847 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:25.138382 (+ 1535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:25.138390 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:25.138396 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:25.138544 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:25.139400 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:25.139406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:25.139409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:25.139740 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:25.139874 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:25.140709 (+ 835us) server_negotiation.cc:299] Negotiation successful
0925 05:01:25.140928 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11783,"thread_start_us":11668,"threads_started":1}
I20260925 05:01:25.142876 6672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:25.101586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:41659 (local address 127.30.247.3:55505)
0925 05:01:25.102977 (+ 1391us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:25.103043 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:25.107819 (+ 4776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:25.113321 (+ 5502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:25.113326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:25.113722 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:25.113987 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:25.113997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.115391 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.115396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:25.116871 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:25.116883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.117982 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.117987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:25.118185 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:25.119475 (+ 1290us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:25.119512 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:25.132472 (+ 12960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:25.137043 (+ 4571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:25.137047 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:25.137050 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:25.137583 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:25.138727 (+ 1144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:25.138731 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:25.138734 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:25.138928 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:25.139989 (+ 1061us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:25.139993 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:25.140136 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 05:01:25.141543 (+ 1407us) client_negotiation.cc:240] Negotiation successful
0925 05:01:25.141826 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1026,"mutex_wait_us":608,"thread_start_us":771,"threads_started":1}
I20260925 05:01:25.143816 6665 heartbeater.cc:347] Connected to a master server at 127.30.247.60:41659
I20260925 05:01:25.143942 6665 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:25.144280 6670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:25.101251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:53319 (local address 127.30.247.62:44265)
0925 05:01:25.101851 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:25.101860 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:25.112250 (+ 10390us) server_negotiation.cc:407] Connection header received
0925 05:01:25.113354 (+ 1104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:25.113363 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:25.113485 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:25.113591 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:25.115407 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.119089 (+ 3682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.121795 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.122760 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.122890 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:25.136338 (+ 13448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:25.136345 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:25.136352 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:25.136401 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:25.140674 (+ 4273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:25.141710 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:25.141715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:25.141719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:25.141874 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:25.142441 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:25.142446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:25.142449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:25.142796 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:25.143182 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:25.143688 (+ 506us) server_negotiation.cc:299] Negotiation successful
0925 05:01:25.143903 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":531,"threads_started":1}
I20260925 05:01:25.144351 6665 heartbeater.cc:511] Master 127.30.247.60:41659 requested a full tablet report, sending...
I20260925 05:01:25.145418 6204 ts_manager.cc:194] Registered new tserver with Master: ad8603b082d240c287b1cf61856b67ce (127.30.247.3:41837)
I20260925 05:01:25.145577 6675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:25.102145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.30.247.3:53319)
0925 05:01:25.112110 (+ 9965us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:25.112139 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:25.113166 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:25.113886 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:25.113894 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:25.114712 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:25.115182 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:25.115199 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.119288 (+ 4089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.119294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:25.121548 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:25.121570 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.123263 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.123272 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:25.123562 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:25.124216 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:25.124242 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:25.136136 (+ 11894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:25.140860 (+ 4724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:25.140865 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:25.140869 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:25.141589 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:25.142053 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:25.142061 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:25.142067 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:25.142280 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:25.143263 (+ 983us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:25.143269 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:25.143406 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 05:01:25.144934 (+ 1528us) client_negotiation.cc:240] Negotiation successful
0925 05:01:25.145189 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9780,"mutex_wait_us":4,"thread_start_us":9581,"threads_started":1}
I20260925 05:01:25.146418 6666 heartbeater.cc:347] Connected to a master server at 127.30.247.62:44265
I20260925 05:01:25.146579 6666 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:25.147038 6666 heartbeater.cc:511] Master 127.30.247.62:44265 requested a full tablet report, sending...
I20260925 05:01:25.148010 5412 ts_manager.cc:194] Re-registered known tserver with Master: ad8603b082d240c287b1cf61856b67ce (127.30.247.3:41837)
I20260925 05:01:25.149580 5412 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.247.3@KRBTEST.COM'} at 127.30.247.3:53319
I20260925 05:01:25.156183 6683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:25.129904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40034 (local address 127.30.247.61:42081)
0925 05:01:25.131070 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:25.131076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:25.133945 (+ 2869us) server_negotiation.cc:407] Connection header received
0925 05:01:25.134002 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:25.134007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:25.134076 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:25.134180 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:25.138041 (+ 3861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.139252 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.141238 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.141943 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.142081 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:25.148206 (+ 6125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:25.148213 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:25.148218 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:25.148260 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:25.153329 (+ 5069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:25.154240 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:25.154245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:25.154249 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:25.154396 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:25.154868 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:25.154874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:25.154877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:25.155149 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:25.155399 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:25.155678 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 05:01:25.155868 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1028,"thread_start_us":865,"threads_started":1}
I20260925 05:01:25.161638 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:25.124809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54776 (local address 127.30.247.60:41659)
0925 05:01:25.129187 (+ 4378us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:25.129196 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:25.133733 (+ 4537us) server_negotiation.cc:407] Connection header received
0925 05:01:25.133802 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:25.133808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:25.133896 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:25.134012 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:25.142453 (+ 8441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.143819 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.151106 (+ 7287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.151671 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.151783 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:25.155016 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:25.155026 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:25.155034 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:25.155070 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:25.158459 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:25.159218 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:25.159222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:25.159224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:25.159352 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:25.159739 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:25.159745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:25.159748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:25.160107 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:25.160329 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:25.161018 (+ 689us) server_negotiation.cc:299] Negotiation successful
0925 05:01:25.161280 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4198,"thread_start_us":4056,"threads_started":1}
I20260925 05:01:25.162505 6670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:25.120052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49728 (local address 127.30.247.62:44265)
0925 05:01:25.144580 (+ 24528us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:25.144586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:25.144604 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:01:25.144648 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:25.144652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:25.144723 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:25.144821 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:25.147224 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.148859 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.150648 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.151268 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.151415 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:25.154839 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:25.154846 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:25.154851 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:25.154915 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:25.159361 (+ 4446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:25.160215 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:25.160218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:25.160220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:25.160329 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:25.160765 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:25.160768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:25.160770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:25.161134 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:25.161401 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:25.161984 (+ 583us) server_negotiation.cc:299] Negotiation successful
0925 05:01:25.162173 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24414,"thread_start_us":28536,"threads_started":1}
I20260925 05:01:25.166471 5412 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:01:25.174599 5412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:01:25.220805 6696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:25.195425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:33901 (local address 127.0.0.1:51150)
0925 05:01:25.196039 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:25.196057 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:25.196168 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:25.212508 (+ 16340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:25.212516 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:25.212573 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:01:25.213083 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:25.213098 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.215110 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.215118 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:25.217935 (+ 2817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:25.217952 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.219869 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.219874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:25.219992 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:25.220057 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 05:01:25.220171 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 05:01:25.220314 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":60,"thread_start_us":290,"threads_started":1}
I20260925 05:01:25.221050 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:25.196440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54682 (local address 127.30.247.2:43559)
0925 05:01:25.205405 (+ 8965us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:25.205416 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:25.210521 (+ 5105us) server_negotiation.cc:407] Connection header received
0925 05:01:25.211048 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:25.211062 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:25.211444 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:25.211649 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:01:25.212697 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.215246 (+ 2549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.218636 (+ 3390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.219946 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.220082 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:25.220283 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:25.220521 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 05:01:25.220724 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8802,"thread_start_us":8683,"threads_started":1}
I20260925 05:01:25.220988 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:25.196114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:43559 (local address 127.0.0.1:54682)
0925 05:01:25.210359 (+ 14245us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:25.210386 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:25.210567 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:25.211999 (+ 1432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:25.212004 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:25.212031 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:01:25.212459 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:25.212474 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.215401 (+ 2927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.215405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:25.218495 (+ 3090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:25.218507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.220155 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.220163 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:25.220326 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:25.220356 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 05:01:25.220441 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 05:01:25.220591 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14073,"mutex_wait_us":752,"thread_start_us":20312,"threads_started":1}
I20260925 05:01:25.222383 6697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:25.195718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51150 (local address 127.30.247.1:33901)
0925 05:01:25.211847 (+ 16129us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:25.211859 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:25.211891 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:01:25.212001 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:25.212009 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:25.212273 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:25.212507 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:01:25.213320 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.214977 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.218185 (+ 3208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.219669 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.219808 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:25.220029 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:25.221875 (+ 1846us) server_negotiation.cc:299] Negotiation successful
0925 05:01:25.222065 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15942,"thread_start_us":16968,"threads_started":1}
I20260925 05:01:25.225265 6459 tablet_service.cc:1467] Processing CreateTablet for tablet 7c02bd4f5d374289bcd0d63535b7a072 (DEFAULT_TABLE table=test-table [id=fe1a9beeb71949f385d04aa1e80bdd9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:01:25.226161 6698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:25.198406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:41837 (local address 127.0.0.1:60748)
0925 05:01:25.216738 (+ 18332us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:25.216756 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:25.217069 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:25.217749 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:25.217753 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:25.217781 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:01:25.218291 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:25.218303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.221799 (+ 3496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.221807 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:25.223891 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:25.223905 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.225390 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.225396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:25.225568 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:25.225603 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 05:01:25.225715 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 05:01:25.225846 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18181,"thread_start_us":12288,"threads_started":1}
I20260925 05:01:25.226212 6701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:25.198739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60748 (local address 127.30.247.3:41837)
0925 05:01:25.216388 (+ 17649us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:25.216397 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:25.217206 (+ 809us) server_negotiation.cc:407] Connection header received
0925 05:01:25.217308 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:25.217332 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:25.217562 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:25.217760 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:01:25.220140 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.221672 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.224093 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.225199 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.225361 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:25.225552 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:25.225764 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 05:01:25.225905 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17469,"thread_start_us":17271,"threads_started":1}
I20260925 05:01:25.226099 6319 tablet_service.cc:1467] Processing CreateTablet for tablet 7c02bd4f5d374289bcd0d63535b7a072 (DEFAULT_TABLE table=test-table [id=fe1a9beeb71949f385d04aa1e80bdd9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:01:25.226835 6459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c02bd4f5d374289bcd0d63535b7a072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:25.227476 6319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c02bd4f5d374289bcd0d63535b7a072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:25.231422 6600 tablet_service.cc:1467] Processing CreateTablet for tablet 7c02bd4f5d374289bcd0d63535b7a072 (DEFAULT_TABLE table=test-table [id=fe1a9beeb71949f385d04aa1e80bdd9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:01:25.232951 6600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c02bd4f5d374289bcd0d63535b7a072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:25.243984 6702 tablet_bootstrap.cc:492] T 7c02bd4f5d374289bcd0d63535b7a072 P ad8603b082d240c287b1cf61856b67ce: Bootstrap starting.
I20260925 05:01:25.246987 6703 tablet_bootstrap.cc:492] T 7c02bd4f5d374289bcd0d63535b7a072 P 41fed7f820404f16aa2d506396ef227c: Bootstrap starting.
I20260925 05:01:25.247335 6702 tablet_bootstrap.cc:654] T 7c02bd4f5d374289bcd0d63535b7a072 P ad8603b082d240c287b1cf61856b67ce: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:25.248272 6704 tablet_bootstrap.cc:492] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426: Bootstrap starting.
I20260925 05:01:25.248631 6702 log.cc:824] T 7c02bd4f5d374289bcd0d63535b7a072 P ad8603b082d240c287b1cf61856b67ce: Log is configured to *not* fsync() on all Append() calls
I20260925 05:01:25.251271 6703 tablet_bootstrap.cc:654] T 7c02bd4f5d374289bcd0d63535b7a072 P 41fed7f820404f16aa2d506396ef227c: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:25.251312 6702 tablet_bootstrap.cc:492] T 7c02bd4f5d374289bcd0d63535b7a072 P ad8603b082d240c287b1cf61856b67ce: No bootstrap required, opened a new log
I20260925 05:01:25.251577 6702 ts_tablet_manager.cc:1397] T 7c02bd4f5d374289bcd0d63535b7a072 P ad8603b082d240c287b1cf61856b67ce: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 05:01:25.252207 6704 tablet_bootstrap.cc:654] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:25.252645 6703 log.cc:824] T 7c02bd4f5d374289bcd0d63535b7a072 P 41fed7f820404f16aa2d506396ef227c: Log is configured to *not* fsync() on all Append() calls
I20260925 05:01:25.253481 6704 log.cc:824] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426: Log is configured to *not* fsync() on all Append() calls
I20260925 05:01:25.256184 6703 tablet_bootstrap.cc:492] T 7c02bd4f5d374289bcd0d63535b7a072 P 41fed7f820404f16aa2d506396ef227c: No bootstrap required, opened a new log
I20260925 05:01:25.256664 6703 ts_tablet_manager.cc:1397] T 7c02bd4f5d374289bcd0d63535b7a072 P 41fed7f820404f16aa2d506396ef227c: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260925 05:01:25.259021 6704 tablet_bootstrap.cc:492] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426: No bootstrap required, opened a new log
I20260925 05:01:25.259289 6704 ts_tablet_manager.cc:1397] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260925 05:01:25.259542 6702 raft_consensus.cc:348] T 7c02bd4f5d374289bcd0d63535b7a072 P ad8603b082d240c287b1cf61856b67ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fed7f820404f16aa2d506396ef227c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33901 } } peers { permanent_uuid: "87f2a171c1c8411e93504e0b8a47b426" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 43559 } } peers { permanent_uuid: "ad8603b082d240c287b1cf61856b67ce" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41837 } }
I20260925 05:01:25.259879 6702 raft_consensus.cc:374] T 7c02bd4f5d374289bcd0d63535b7a072 P ad8603b082d240c287b1cf61856b67ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:25.260056 6702 raft_consensus.cc:729] T 7c02bd4f5d374289bcd0d63535b7a072 P ad8603b082d240c287b1cf61856b67ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad8603b082d240c287b1cf61856b67ce, State: Initialized, Role: FOLLOWER
I20260925 05:01:25.260749 6702 consensus_queue.cc:260] T 7c02bd4f5d374289bcd0d63535b7a072 P ad8603b082d240c287b1cf61856b67ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fed7f820404f16aa2d506396ef227c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33901 } } peers { permanent_uuid: "87f2a171c1c8411e93504e0b8a47b426" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 43559 } } peers { permanent_uuid: "ad8603b082d240c287b1cf61856b67ce" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41837 } }
I20260925 05:01:25.262413 6702 ts_tablet_manager.cc:1428] T 7c02bd4f5d374289bcd0d63535b7a072 P ad8603b082d240c287b1cf61856b67ce: Time spent starting tablet: real 0.011s user 0.003s sys 0.008s
I20260925 05:01:25.262662 6666 heartbeater.cc:503] Master 127.30.247.62:44265 was elected leader, sending a full tablet report...
I20260925 05:01:25.268450 6703 raft_consensus.cc:348] T 7c02bd4f5d374289bcd0d63535b7a072 P 41fed7f820404f16aa2d506396ef227c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fed7f820404f16aa2d506396ef227c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33901 } } peers { permanent_uuid: "87f2a171c1c8411e93504e0b8a47b426" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 43559 } } peers { permanent_uuid: "ad8603b082d240c287b1cf61856b67ce" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41837 } }
I20260925 05:01:25.268882 6703 raft_consensus.cc:374] T 7c02bd4f5d374289bcd0d63535b7a072 P 41fed7f820404f16aa2d506396ef227c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:25.268963 6703 raft_consensus.cc:729] T 7c02bd4f5d374289bcd0d63535b7a072 P 41fed7f820404f16aa2d506396ef227c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41fed7f820404f16aa2d506396ef227c, State: Initialized, Role: FOLLOWER
I20260925 05:01:25.269430 6703 consensus_queue.cc:260] T 7c02bd4f5d374289bcd0d63535b7a072 P 41fed7f820404f16aa2d506396ef227c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fed7f820404f16aa2d506396ef227c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33901 } } peers { permanent_uuid: "87f2a171c1c8411e93504e0b8a47b426" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 43559 } } peers { permanent_uuid: "ad8603b082d240c287b1cf61856b67ce" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41837 } }
I20260925 05:01:25.271265 6703 ts_tablet_manager.cc:1428] T 7c02bd4f5d374289bcd0d63535b7a072 P 41fed7f820404f16aa2d506396ef227c: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260925 05:01:25.272359 6704 raft_consensus.cc:348] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fed7f820404f16aa2d506396ef227c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33901 } } peers { permanent_uuid: "87f2a171c1c8411e93504e0b8a47b426" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 43559 } } peers { permanent_uuid: "ad8603b082d240c287b1cf61856b67ce" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41837 } }
I20260925 05:01:25.272771 6704 raft_consensus.cc:374] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:25.272866 6704 raft_consensus.cc:729] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f2a171c1c8411e93504e0b8a47b426, State: Initialized, Role: FOLLOWER
I20260925 05:01:25.273469 6704 consensus_queue.cc:260] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fed7f820404f16aa2d506396ef227c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33901 } } peers { permanent_uuid: "87f2a171c1c8411e93504e0b8a47b426" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 43559 } } peers { permanent_uuid: "ad8603b082d240c287b1cf61856b67ce" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41837 } }
I20260925 05:01:25.275981 6525 heartbeater.cc:503] Master 127.30.247.62:44265 was elected leader, sending a full tablet report...
I20260925 05:01:25.276261 6704 ts_tablet_manager.cc:1428] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426: Time spent starting tablet: real 0.017s user 0.013s sys 0.002s
W20260925 05:01:25.283237 6527 tablet.cc:2369] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:01:25.362879 6677 tablet.cc:2369] T 7c02bd4f5d374289bcd0d63535b7a072 P ad8603b082d240c287b1cf61856b67ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:01:25.468639 6387 tablet.cc:2369] T 7c02bd4f5d374289bcd0d63535b7a072 P 41fed7f820404f16aa2d506396ef227c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:01:25.484913 6710 raft_consensus.cc:482] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:01:25.485173 6710 raft_consensus.cc:504] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fed7f820404f16aa2d506396ef227c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33901 } } peers { permanent_uuid: "87f2a171c1c8411e93504e0b8a47b426" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 43559 } } peers { permanent_uuid: "ad8603b082d240c287b1cf61856b67ce" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41837 } }
I20260925 05:01:25.487325 6710 leader_election.cc:288] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41fed7f820404f16aa2d506396ef227c (127.30.247.1:33901), ad8603b082d240c287b1cf61856b67ce (127.30.247.3:41837)
I20260925 05:01:25.493903 6701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:25.487991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:41539 (local address 127.30.247.3:41837)
0925 05:01:25.488197 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:25.488203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:25.488516 (+ 313us) server_negotiation.cc:407] Connection header received
0925 05:01:25.488732 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:25.488737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:25.488814 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:25.488926 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:01:25.489469 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.490156 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.492124 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.492986 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.493112 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:25.493162 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:25.493527 (+ 365us) server_negotiation.cc:299] Negotiation successful
0925 05:01:25.493659 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 05:01:25.493964 6697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:25.487590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:33755 (local address 127.30.247.1:33901)
0925 05:01:25.487816 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:25.487823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:25.488304 (+ 481us) server_negotiation.cc:407] Connection header received
0925 05:01:25.488499 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:25.488506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:25.488612 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:25.488708 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:01:25.489319 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.490075 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.492197 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.493135 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.493284 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:25.493352 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:25.493625 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 05:01:25.493753 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 05:01:25.494127 6712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:25.487554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:41837 (local address 127.30.247.2:41539)
0925 05:01:25.488391 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:25.488426 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:25.488573 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:25.488946 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:25.488949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:25.488968 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:01:25.489306 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:25.489317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.490287 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.490290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:25.491977 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:25.491988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.493173 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.493181 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:25.493346 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:25.493387 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 05:01:25.493469 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 05:01:25.493583 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":22,"thread_start_us":538,"threads_started":1}
I20260925 05:01:25.494350 6711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:25.487481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:33901 (local address 127.30.247.2:33755)
0925 05:01:25.488185 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:25.488206 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:25.488347 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:25.488730 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:25.488735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:25.488761 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:01:25.489178 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:25.489188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.490202 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.490209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:25.492026 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:25.492037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.493328 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.493333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:25.493438 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:25.493492 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 05:01:25.493583 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 05:01:25.493703 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":53,"thread_start_us":293,"threads_started":1}
I20260925 05:01:25.495138 6620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c02bd4f5d374289bcd0d63535b7a072" candidate_uuid: "87f2a171c1c8411e93504e0b8a47b426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad8603b082d240c287b1cf61856b67ce" is_pre_election: true
I20260925 05:01:25.495220 6339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c02bd4f5d374289bcd0d63535b7a072" candidate_uuid: "87f2a171c1c8411e93504e0b8a47b426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41fed7f820404f16aa2d506396ef227c" is_pre_election: true
I20260925 05:01:25.495625 6620 raft_consensus.cc:2454] T 7c02bd4f5d374289bcd0d63535b7a072 P ad8603b082d240c287b1cf61856b67ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87f2a171c1c8411e93504e0b8a47b426 in term 0.
I20260925 05:01:25.495671 6339 raft_consensus.cc:2454] T 7c02bd4f5d374289bcd0d63535b7a072 P 41fed7f820404f16aa2d506396ef227c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87f2a171c1c8411e93504e0b8a47b426 in term 0.
I20260925 05:01:25.496266 6412 leader_election.cc:302] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41fed7f820404f16aa2d506396ef227c, 87f2a171c1c8411e93504e0b8a47b426; no voters:
I20260925 05:01:25.496654 6710 raft_consensus.cc:2781] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:01:25.496797 6710 raft_consensus.cc:482] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:01:25.496901 6710 raft_consensus.cc:3037] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:25.499048 6710 raft_consensus.cc:504] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fed7f820404f16aa2d506396ef227c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33901 } } peers { permanent_uuid: "87f2a171c1c8411e93504e0b8a47b426" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 43559 } } peers { permanent_uuid: "ad8603b082d240c287b1cf61856b67ce" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41837 } }
I20260925 05:01:25.499641 6710 leader_election.cc:288] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426 [CANDIDATE]: Term 1 election: Requested vote from peers 41fed7f820404f16aa2d506396ef227c (127.30.247.1:33901), ad8603b082d240c287b1cf61856b67ce (127.30.247.3:41837)
I20260925 05:01:25.499975 6339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c02bd4f5d374289bcd0d63535b7a072" candidate_uuid: "87f2a171c1c8411e93504e0b8a47b426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41fed7f820404f16aa2d506396ef227c"
I20260925 05:01:25.500084 6620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c02bd4f5d374289bcd0d63535b7a072" candidate_uuid: "87f2a171c1c8411e93504e0b8a47b426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad8603b082d240c287b1cf61856b67ce"
I20260925 05:01:25.500145 6339 raft_consensus.cc:3037] T 7c02bd4f5d374289bcd0d63535b7a072 P 41fed7f820404f16aa2d506396ef227c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:25.500277 6620 raft_consensus.cc:3037] T 7c02bd4f5d374289bcd0d63535b7a072 P ad8603b082d240c287b1cf61856b67ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:25.502269 6339 raft_consensus.cc:2454] T 7c02bd4f5d374289bcd0d63535b7a072 P 41fed7f820404f16aa2d506396ef227c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87f2a171c1c8411e93504e0b8a47b426 in term 1.
I20260925 05:01:25.502408 6620 raft_consensus.cc:2454] T 7c02bd4f5d374289bcd0d63535b7a072 P ad8603b082d240c287b1cf61856b67ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87f2a171c1c8411e93504e0b8a47b426 in term 1.
I20260925 05:01:25.502628 6412 leader_election.cc:302] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41fed7f820404f16aa2d506396ef227c, 87f2a171c1c8411e93504e0b8a47b426; no voters:
I20260925 05:01:25.502903 6710 raft_consensus.cc:2781] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:01:25.503324 6710 raft_consensus.cc:686] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426 [term 1 LEADER]: Becoming Leader. State: Replica: 87f2a171c1c8411e93504e0b8a47b426, State: Running, Role: LEADER
I20260925 05:01:25.503813 6710 consensus_queue.cc:237] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fed7f820404f16aa2d506396ef227c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33901 } } peers { permanent_uuid: "87f2a171c1c8411e93504e0b8a47b426" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 43559 } } peers { permanent_uuid: "ad8603b082d240c287b1cf61856b67ce" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41837 } }
I20260925 05:01:25.507984 5412 catalog_manager.cc:5495] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87f2a171c1c8411e93504e0b8a47b426 (127.30.247.2). New cstate: current_term: 1 leader_uuid: "87f2a171c1c8411e93504e0b8a47b426" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fed7f820404f16aa2d506396ef227c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87f2a171c1c8411e93504e0b8a47b426" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 43559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad8603b082d240c287b1cf61856b67ce" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41837 } health_report { overall_health: UNKNOWN } } }
I20260925 05:01:25.569350 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:25.564312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54684 (local address 127.30.247.2:43559)
0925 05:01:25.564487 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:25.564491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:25.564509 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:01:25.564694 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:25.564697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:25.564771 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:25.564894 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:01:25.565236 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.565948 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.567288 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.567670 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.567762 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:25.568300 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:01:25.568868 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:01:25.568992 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:25.569091 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 05:01:25.569152 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 05:01:25.577445 6620 raft_consensus.cc:1261] T 7c02bd4f5d374289bcd0d63535b7a072 P ad8603b082d240c287b1cf61856b67ce [term 1 FOLLOWER]: Refusing update from remote peer 87f2a171c1c8411e93504e0b8a47b426: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:01:25.577448 6339 raft_consensus.cc:1261] T 7c02bd4f5d374289bcd0d63535b7a072 P 41fed7f820404f16aa2d506396ef227c [term 1 FOLLOWER]: Refusing update from remote peer 87f2a171c1c8411e93504e0b8a47b426: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:01:25.578475 6713 consensus_queue.cc:1035] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad8603b082d240c287b1cf61856b67ce" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:01:25.578811 6710 consensus_queue.cc:1035] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41fed7f820404f16aa2d506396ef227c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:01:25.594290 6717 mvcc.cc:205] Tried to move back new op lower bound from 7333119940915793920 to 7333119940627443712. Current Snapshot: MvccSnapshot[applied={T|T < 7333119940915793920}]
I20260925 05:01:25.607810 6718 mvcc.cc:205] Tried to move back new op lower bound from 7333119940915793920 to 7333119940627443712. Current Snapshot: MvccSnapshot[applied={T|T < 7333119940915793920}]
I20260925 05:01:25.614938 5412 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:01:25.615188 5412 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:01:25.620739 5412 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 190a829b060c49298dcf7111c5dab8c6: Sending DeleteTablet for 3 replicas of tablet 7c02bd4f5d374289bcd0d63535b7a072
I20260925 05:01:25.622066 6459 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c02bd4f5d374289bcd0d63535b7a072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:01:25 UTC) from {username='oryx', principal='oryx/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:54682
I20260925 05:01:25.622080 6319 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c02bd4f5d374289bcd0d63535b7a072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:01:25 UTC) from {username='oryx', principal='oryx/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:51150
I20260925 05:01:25.622699 6600 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c02bd4f5d374289bcd0d63535b7a072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:01:25 UTC) from {username='oryx', principal='oryx/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:60748
I20260925 05:01:25.632858 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 5380
I20260925 05:01:25.634132 5442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:25.647614 6729 tablet_replica.cc:330] T 7c02bd4f5d374289bcd0d63535b7a072 P ad8603b082d240c287b1cf61856b67ce: stopping tablet replica
I20260925 05:01:25.648080 6729 raft_consensus.cc:2229] T 7c02bd4f5d374289bcd0d63535b7a072 P ad8603b082d240c287b1cf61856b67ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:01:25.648767 6727 tablet_replica.cc:330] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426: stopping tablet replica
I20260925 05:01:25.649493 6727 raft_consensus.cc:2229] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:01:25.650132 6727 raft_consensus.cc:2258] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:01:25.650727 6729 raft_consensus.cc:2258] T 7c02bd4f5d374289bcd0d63535b7a072 P ad8603b082d240c287b1cf61856b67ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:01:25.652989 6729 ts_tablet_manager.cc:1905] T 7c02bd4f5d374289bcd0d63535b7a072 P ad8603b082d240c287b1cf61856b67ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:01:25.658033 6727 ts_tablet_manager.cc:1905] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:01:25.659739 6728 tablet_replica.cc:330] T 7c02bd4f5d374289bcd0d63535b7a072 P 41fed7f820404f16aa2d506396ef227c: stopping tablet replica
I20260925 05:01:25.660202 6728 raft_consensus.cc:2229] T 7c02bd4f5d374289bcd0d63535b7a072 P 41fed7f820404f16aa2d506396ef227c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:01:25.660723 6728 raft_consensus.cc:2258] T 7c02bd4f5d374289bcd0d63535b7a072 P 41fed7f820404f16aa2d506396ef227c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:01:25.662845 6728 ts_tablet_manager.cc:1905] T 7c02bd4f5d374289bcd0d63535b7a072 P 41fed7f820404f16aa2d506396ef227c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:01:25.667073 6729 ts_tablet_manager.cc:1918] T 7c02bd4f5d374289bcd0d63535b7a072 P ad8603b082d240c287b1cf61856b67ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:01:25.667670 6683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:25.624990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40040 (local address 127.30.247.61:42081)
0925 05:01:25.625188 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:25.625196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:25.631531 (+ 6335us) server_negotiation.cc:407] Connection header received
0925 05:01:25.632032 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:25.632053 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:25.632144 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:25.632222 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:25.643808 (+ 11586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.645184 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.647682 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.648286 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.648400 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:25.660744 (+ 12344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:25.660751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:25.660758 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:25.660802 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:25.664358 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:25.665282 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:25.665288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:25.665291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:25.665432 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:25.665845 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:25.665850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:25.665854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:25.666197 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:25.666418 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:25.666993 (+ 575us) server_negotiation.cc:299] Negotiation successful
0925 05:01:25.667324 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 05:01:25.669595 6728 ts_tablet_manager.cc:1918] T 7c02bd4f5d374289bcd0d63535b7a072 P 41fed7f820404f16aa2d506396ef227c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:01:25.669747 6728 log.cc:1196] T 7c02bd4f5d374289bcd0d63535b7a072 P 41fed7f820404f16aa2d506396ef227c: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/wal/wals/7c02bd4f5d374289bcd0d63535b7a072
I20260925 05:01:25.670408 6728 ts_tablet_manager.cc:1939] T 7c02bd4f5d374289bcd0d63535b7a072 P 41fed7f820404f16aa2d506396ef227c: Deleting consensus metadata
I20260925 05:01:25.671520 6727 ts_tablet_manager.cc:1918] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:01:25.671741 6729 log.cc:1196] T 7c02bd4f5d374289bcd0d63535b7a072 P ad8603b082d240c287b1cf61856b67ce: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/wal/wals/7c02bd4f5d374289bcd0d63535b7a072
I20260925 05:01:25.672288 6729 ts_tablet_manager.cc:1939] T 7c02bd4f5d374289bcd0d63535b7a072 P ad8603b082d240c287b1cf61856b67ce: Deleting consensus metadata
I20260925 05:01:25.672472 6727 log.cc:1196] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/wal/wals/7c02bd4f5d374289bcd0d63535b7a072
I20260925 05:01:25.674173 6727 ts_tablet_manager.cc:1939] T 7c02bd4f5d374289bcd0d63535b7a072 P 87f2a171c1c8411e93504e0b8a47b426: Deleting consensus metadata
I20260925 05:01:25.685551 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:25.631625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54784 (local address 127.30.247.60:41659)
0925 05:01:25.631833 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:25.631841 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:25.669960 (+ 38119us) server_negotiation.cc:407] Connection header received
0925 05:01:25.670278 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:25.670287 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:25.670403 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:25.670484 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:25.672394 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.673530 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.674916 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.675513 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.675645 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:25.679086 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:25.679092 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:25.679097 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:25.679135 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:25.682495 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:25.683272 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:25.683277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:25.683281 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:25.683413 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:25.683851 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:25.683856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:25.683859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:25.684203 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:25.684410 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:25.684998 (+ 588us) server_negotiation.cc:299] Negotiation successful
0925 05:01:25.685211 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 05:01:25.767410 5397 catalog_manager.cc:4841] TS ad8603b082d240c287b1cf61856b67ce (127.30.247.3:41837): tablet 7c02bd4f5d374289bcd0d63535b7a072 (table test-table [id=fe1a9beeb71949f385d04aa1e80bdd9c]) successfully deleted
I20260925 05:01:25.768812 5396 catalog_manager.cc:4841] TS 87f2a171c1c8411e93504e0b8a47b426 (127.30.247.2:43559): tablet 7c02bd4f5d374289bcd0d63535b7a072 (table test-table [id=fe1a9beeb71949f385d04aa1e80bdd9c]) successfully deleted
I20260925 05:01:25.769405 5396 catalog_manager.cc:4841] TS 41fed7f820404f16aa2d506396ef227c (127.30.247.1:33901): tablet 7c02bd4f5d374289bcd0d63535b7a072 (table test-table [id=fe1a9beeb71949f385d04aa1e80bdd9c]) successfully deleted
I20260925 05:01:25.770022 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 5380
I20260925 05:01:25.824321 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:25.799345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54786 (local address 127.30.247.60:41659)
0925 05:01:25.799650 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:25.799659 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:25.806309 (+ 6650us) server_negotiation.cc:407] Connection header received
0925 05:01:25.806762 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:25.806768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:25.806849 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:25.806922 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:25.809211 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.810311 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.812299 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.812974 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.813070 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:25.817827 (+ 4757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:25.817835 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:25.817841 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:25.817878 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:25.821297 (+ 3419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:25.822071 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:25.822077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:25.822081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:25.822217 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:25.822651 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:25.822656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:25.822660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:25.822986 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:25.823200 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:25.823757 (+ 557us) server_negotiation.cc:299] Negotiation successful
0925 05:01:25.823962 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260925 05:01:25.827416 6683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:25.799865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40044 (local address 127.30.247.61:42081)
0925 05:01:25.800085 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:25.800093 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:25.810693 (+ 10600us) server_negotiation.cc:407] Connection header received
0925 05:01:25.811252 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:25.811258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:25.811350 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:25.811412 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:25.813916 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.815064 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.816963 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:25.817656 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:25.817739 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:25.820785 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:25.820791 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:25.820796 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:25.820832 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:25.824259 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:25.825063 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:25.825071 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:25.825079 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:25.825211 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:25.825622 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:25.825629 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:25.825633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:25.825942 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:25.826143 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:25.826717 (+ 574us) server_negotiation.cc:299] Negotiation successful
0925 05:01:25.826941 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 05:01:26.604768 6739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:26.603827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.30.247.1:55909)
0925 05:01:26.604379 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:26.604533 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":233,"threads_started":1}
W20260925 05:01:26.605312 6385 heartbeater.cc:644] Failed to heartbeat to 127.30.247.62:44265 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
I20260925 05:01:26.612776 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:26.611974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.30.247.3:57973)
0925 05:01:26.612363 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:26.612519 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":202,"threads_started":1}
W20260925 05:01:26.613173 6666 heartbeater.cc:644] Failed to heartbeat to 127.30.247.62:44265 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
I20260925 05:01:26.615613 6741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:26.615008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.30.247.2:55789)
0925 05:01:26.615372 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:26.615478 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":180,"threads_started":1}
W20260925 05:01:26.615964 6525 heartbeater.cc:644] Failed to heartbeat to 127.30.247.62:44265 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
I20260925 05:01:27.122332 6742 raft_consensus.cc:482] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 190a829b060c49298dcf7111c5dab8c6)
I20260925 05:01:27.122601 6742 raft_consensus.cc:504] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: false } }
I20260925 05:01:27.122864 6743 raft_consensus.cc:482] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 190a829b060c49298dcf7111c5dab8c6)
I20260925 05:01:27.123131 6743 raft_consensus.cc:504] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: false } }
I20260925 05:01:27.123545 6742 leader_election.cc:288] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 190a829b060c49298dcf7111c5dab8c6 (127.30.247.62:44265), 543dab666dc748caa9542055c3b85e43 (127.30.247.61:42081)
I20260925 05:01:27.133558 6743 leader_election.cc:288] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 190a829b060c49298dcf7111c5dab8c6 (127.30.247.62:44265), b27f6d41f6f1426bb70abaf6985e00b2 (127.30.247.60:41659)
I20260925 05:01:27.134361 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.123789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.0.0.1:50190)
0925 05:01:27.133781 (+ 9992us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.134028 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9872,"threads_started":1}
I20260925 05:01:27.134856 6748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.124984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.0.0.1:50198)
0925 05:01:27.133462 (+ 8478us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.134364 (+ 902us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8291,"threads_started":1}
W20260925 05:01:27.137918 6188 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
W20260925 05:01:27.139779 5994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
I20260925 05:01:27.142875 6746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.123902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40058 (local address 127.30.247.61:42081)
0925 05:01:27.133686 (+ 9784us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:27.134213 (+ 527us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:27.134943 (+ 730us) server_negotiation.cc:407] Connection header received
0925 05:01:27.135032 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:27.135040 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:27.135136 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:27.135241 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:01:27.136135 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:27.137623 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:27.140222 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:27.141612 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:27.141720 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:27.141807 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:27.142251 (+ 444us) server_negotiation.cc:299] Negotiation successful
0925 05:01:27.142457 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":9649,"thread_start_us":9447,"threads_started":1}
I20260925 05:01:27.143748 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.123853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:42081 (local address 127.0.0.1:40058)
0925 05:01:27.134596 (+ 10743us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.134618 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:27.134789 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:27.135524 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:27.135532 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:27.135568 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:01:27.135904 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:27.135920 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:27.137776 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:27.137781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:27.140037 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:27.140050 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:27.141926 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:27.141932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:27.142080 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:27.142112 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 05:01:27.143218 (+ 1106us) client_negotiation.cc:240] Negotiation successful
0925 05:01:27.143368 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10622,"mutex_wait_us":18,"thread_start_us":9875,"threads_started":1}
I20260925 05:01:27.145153 6019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "543dab666dc748caa9542055c3b85e43" is_pre_election: true
I20260925 05:01:27.145406 6019 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b27f6d41f6f1426bb70abaf6985e00b2 in term 1.
I20260925 05:01:27.145918 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.145327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.0.0.1:50226)
0925 05:01:27.145523 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.145696 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 05:01:27.145926 6748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.131131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:41659 (local address 127.0.0.1:54802)
0925 05:01:27.135458 (+ 4327us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.135492 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:27.135638 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:27.137968 (+ 2330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:27.137973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:27.138000 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:01:27.138305 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:27.138316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:27.140296 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:27.140302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:27.143191 (+ 2889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:27.143207 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:27.145009 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:27.145016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:27.145242 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:27.145275 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 05:01:27.145375 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 05:01:27.145528 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4203,"mutex_wait_us":853,"thread_start_us":2006,"threads_started":1}
W20260925 05:01:27.146169 5994 leader_election.cc:334] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 190a829b060c49298dcf7111c5dab8c6 (127.30.247.62:44265): Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
I20260925 05:01:27.146181 6191 leader_election.cc:302] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 543dab666dc748caa9542055c3b85e43, b27f6d41f6f1426bb70abaf6985e00b2; no voters:
I20260925 05:01:27.146440 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.146020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.0.0.1:50232)
0925 05:01:27.146217 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.146303 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 05:01:27.146567 6742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260925 05:01:27.146622 6188 leader_election.cc:334] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 190a829b060c49298dcf7111c5dab8c6 (127.30.247.62:44265): Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
I20260925 05:01:27.146696 6742 raft_consensus.cc:482] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 190a829b060c49298dcf7111c5dab8c6)
I20260925 05:01:27.146873 6742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 1 FOLLOWER]: Advancing to term 2
I20260925 05:01:27.147113 6749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.133973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54802 (local address 127.30.247.60:41659)
0925 05:01:27.137521 (+ 3548us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:27.137526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:27.137543 (+ 17us) server_negotiation.cc:407] Connection header received
0925 05:01:27.137600 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:27.137604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:27.137669 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:27.137764 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:01:27.138648 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:27.140153 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:27.143362 (+ 3209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:27.144667 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:27.144798 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:27.144894 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:27.145993 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0925 05:01:27.146117 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":249,"server-negotiator.queue_time_us":3412,"thread_start_us":3847,"threads_started":1}
I20260925 05:01:27.147620 6214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "543dab666dc748caa9542055c3b85e43" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" is_pre_election: true
I20260925 05:01:27.149226 6742 raft_consensus.cc:504] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: false } }
I20260925 05:01:27.149685 6214 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 543dab666dc748caa9542055c3b85e43 in current term 2: Already voted for candidate b27f6d41f6f1426bb70abaf6985e00b2 in this term.
I20260925 05:01:27.149932 6742 leader_election.cc:288] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [CANDIDATE]: Term 2 election: Requested vote from peers 190a829b060c49298dcf7111c5dab8c6 (127.30.247.62:44265), 543dab666dc748caa9542055c3b85e43 (127.30.247.61:42081)
I20260925 05:01:27.150403 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.149989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.0.0.1:50238)
0925 05:01:27.150168 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.150283 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260925 05:01:27.150498 6019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "543dab666dc748caa9542055c3b85e43"
I20260925 05:01:27.150547 5995 leader_election.cc:302] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 543dab666dc748caa9542055c3b85e43; no voters: 190a829b060c49298dcf7111c5dab8c6, b27f6d41f6f1426bb70abaf6985e00b2
I20260925 05:01:27.150733 6019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [term 1 FOLLOWER]: Advancing to term 2
I20260925 05:01:27.151450 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.151066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.0.0.1:50246)
0925 05:01:27.151231 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.151309 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 05:01:27.151643 6188 leader_election.cc:334] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 190a829b060c49298dcf7111c5dab8c6 (127.30.247.62:44265): Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
I20260925 05:01:27.153098 6019 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b27f6d41f6f1426bb70abaf6985e00b2 in term 2.
I20260925 05:01:27.153298 6743 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 05:01:27.153429 6191 leader_election.cc:302] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 543dab666dc748caa9542055c3b85e43, b27f6d41f6f1426bb70abaf6985e00b2; no voters: 190a829b060c49298dcf7111c5dab8c6
I20260925 05:01:27.153735 6742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 05:01:27.154125 6742 raft_consensus.cc:686] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [term 2 LEADER]: Becoming Leader. State: Replica: b27f6d41f6f1426bb70abaf6985e00b2, State: Running, Role: LEADER
I20260925 05:01:27.154476 6742 consensus_queue.cc:237] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: false } }
I20260925 05:01:27.155918 6752 sys_catalog.cc:455] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b27f6d41f6f1426bb70abaf6985e00b2. Latest consensus state: current_term: 2 leader_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: false } } }
I20260925 05:01:27.156230 6752 sys_catalog.cc:458] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:27.156883 6754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:01:27.160138 6754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:01:27.161027 6754 catalog_manager.cc:1259] Loaded cluster ID: 69f5d87d0fd1476db9d81f6d4c7fdfba
I20260925 05:01:27.161123 6754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:01:27.162263 6754 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:01:27.163090 6754 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2: Loaded TSK: 0
I20260925 05:01:27.163894 6754 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:01:27.204701 6204 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:01:27.212486 6204 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:01:27.216251 6019 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [term 2 FOLLOWER]: Refusing update from remote peer b27f6d41f6f1426bb70abaf6985e00b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260925 05:01:27.216310 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.215824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.0.0.1:50274)
0925 05:01:27.216051 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.216137 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260925 05:01:27.216953 6752 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 05:01:27.217317 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.216903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.0.0.1:50286)
0925 05:01:27.217061 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.217153 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 05:01:27.217463 6188 consensus_peers.cc:487] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 -> Peer 190a829b060c49298dcf7111c5dab8c6 (127.30.247.62:44265): Couldn't send request to peer 190a829b060c49298dcf7111c5dab8c6. Status: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 05:01:27.219549 6743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b27f6d41f6f1426bb70abaf6985e00b2. Latest consensus state: current_term: 2 leader_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: false } } }
I20260925 05:01:27.219805 6743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:01:27.221072 6742 sys_catalog.cc:455] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: false } } }
I20260925 05:01:27.221346 6742 sys_catalog.cc:458] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2 [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:27.222800 6743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "190a829b060c49298dcf7111c5dab8c6" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44265 } } peers { permanent_uuid: "543dab666dc748caa9542055c3b85e43" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 42081 } attrs { promote: false } } peers { permanent_uuid: "b27f6d41f6f1426bb70abaf6985e00b2" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 41659 } attrs { promote: false } } }
I20260925 05:01:27.223065 6743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 543dab666dc748caa9542055c3b85e43 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:01:27.243567 6744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.229313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:43559 (local address 127.0.0.1:54700)
0925 05:01:27.229523 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.229545 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:27.229669 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:27.235715 (+ 6046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:27.235719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:27.235744 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:01:27.236234 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:27.236250 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:27.237799 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:27.237808 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:27.240511 (+ 2703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:27.240527 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:27.242492 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:27.242501 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:27.242788 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:27.242846 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 05:01:27.242923 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 05:01:27.243101 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 05:01:27.244617 6761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.228879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54700 (local address 127.30.247.2:43559)
0925 05:01:27.235441 (+ 6562us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:27.235445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:27.235459 (+ 14us) server_negotiation.cc:407] Connection header received
0925 05:01:27.235515 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:27.235519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:27.235581 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:27.235701 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:01:27.236418 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:27.237608 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:27.240748 (+ 3140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:27.242237 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:27.243976 (+ 1739us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:27.244077 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:27.244143 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 05:01:27.244273 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6459,"thread_start_us":252,"threads_started":1}
I20260925 05:01:27.245381 6459 tablet_service.cc:1467] Processing CreateTablet for tablet d1471c9f15264b62812a9341b75d33f6 (DEFAULT_TABLE table=test-table [id=095fac869fd4434699a612628d4e7b54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:01:27.245848 6459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1471c9f15264b62812a9341b75d33f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:27.247618 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.228408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:33901 (local address 127.0.0.1:51156)
0925 05:01:27.228565 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.228584 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:27.228690 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:27.242055 (+ 13365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:27.242059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:27.242082 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:01:27.242257 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:27.242268 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:27.243544 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:27.243549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:27.245329 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:27.245340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:27.246878 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:27.246887 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:27.247082 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:27.247124 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 05:01:27.247192 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 05:01:27.247331 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 05:01:27.248392 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.228580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51156 (local address 127.30.247.1:33901)
0925 05:01:27.241713 (+ 13133us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:27.241717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:27.241732 (+ 15us) server_negotiation.cc:407] Connection header received
0925 05:01:27.241815 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:27.241824 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:27.241904 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:27.242045 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:01:27.242405 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:27.243376 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:27.245501 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:27.246697 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:27.247836 (+ 1139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:27.247945 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:27.247993 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 05:01:27.248123 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13033,"thread_start_us":12948,"threads_started":1}
I20260925 05:01:27.249265 6319 tablet_service.cc:1467] Processing CreateTablet for tablet d1471c9f15264b62812a9341b75d33f6 (DEFAULT_TABLE table=test-table [id=095fac869fd4434699a612628d4e7b54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:01:27.249697 6319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1471c9f15264b62812a9341b75d33f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:27.251081 6763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.229333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:41837 (local address 127.0.0.1:60762)
0925 05:01:27.240973 (+ 11640us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.240997 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:27.241132 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:27.244414 (+ 3282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:27.244419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:27.244459 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:01:27.244993 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:27.245004 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:27.246369 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:27.246375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:27.248678 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:27.248696 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:27.250311 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:27.250317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:27.250485 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:27.250518 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 05:01:27.250597 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 05:01:27.250717 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11485,"mutex_wait_us":438,"thread_start_us":11389,"threads_started":1}
I20260925 05:01:27.251092 6762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.229462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60762 (local address 127.30.247.3:41837)
0925 05:01:27.244138 (+ 14676us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:27.244144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:27.244161 (+ 17us) server_negotiation.cc:407] Connection header received
0925 05:01:27.244223 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:27.244230 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:27.244297 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:27.244396 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:01:27.245163 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:27.246203 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:27.248863 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:27.250106 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:27.250240 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:27.250332 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:27.250664 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 05:01:27.250803 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14550,"thread_start_us":15064,"threads_started":1}
I20260925 05:01:27.253248 6600 tablet_service.cc:1467] Processing CreateTablet for tablet d1471c9f15264b62812a9341b75d33f6 (DEFAULT_TABLE table=test-table [id=095fac869fd4434699a612628d4e7b54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:01:27.253473 6764 tablet_bootstrap.cc:492] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426: Bootstrap starting.
I20260925 05:01:27.253690 6600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1471c9f15264b62812a9341b75d33f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:27.254861 6764 tablet_bootstrap.cc:654] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:27.256979 6764 tablet_bootstrap.cc:492] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426: No bootstrap required, opened a new log
I20260925 05:01:27.257146 6764 ts_tablet_manager.cc:1397] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 05:01:27.258037 6764 raft_consensus.cc:348] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fed7f820404f16aa2d506396ef227c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33901 } } peers { permanent_uuid: "87f2a171c1c8411e93504e0b8a47b426" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 43559 } } peers { permanent_uuid: "ad8603b082d240c287b1cf61856b67ce" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41837 } }
I20260925 05:01:27.258289 6764 raft_consensus.cc:374] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:27.258378 6764 raft_consensus.cc:729] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f2a171c1c8411e93504e0b8a47b426, State: Initialized, Role: FOLLOWER
I20260925 05:01:27.258602 6764 consensus_queue.cc:260] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fed7f820404f16aa2d506396ef227c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33901 } } peers { permanent_uuid: "87f2a171c1c8411e93504e0b8a47b426" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 43559 } } peers { permanent_uuid: "ad8603b082d240c287b1cf61856b67ce" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41837 } }
I20260925 05:01:27.259586 6764 ts_tablet_manager.cc:1428] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 05:01:27.263173 6524 heartbeater.cc:503] Master 127.30.247.60:41659 was elected leader, sending a full tablet report...
I20260925 05:01:27.265015 6769 tablet_bootstrap.cc:492] T d1471c9f15264b62812a9341b75d33f6 P 41fed7f820404f16aa2d506396ef227c: Bootstrap starting.
I20260925 05:01:27.267547 6769 tablet_bootstrap.cc:654] T d1471c9f15264b62812a9341b75d33f6 P 41fed7f820404f16aa2d506396ef227c: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:27.269356 6768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.259773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.30.247.2:34053)
0925 05:01:27.269039 (+ 9266us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.269164 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9138,"thread_start_us":8995,"threads_started":1}
I20260925 05:01:27.270095 6767 tablet_bootstrap.cc:492] T d1471c9f15264b62812a9341b75d33f6 P ad8603b082d240c287b1cf61856b67ce: Bootstrap starting.
I20260925 05:01:27.270776 6768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.270046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.30.247.2:58401)
0925 05:01:27.270444 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.270575 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248}
I20260925 05:01:27.271901 6767 tablet_bootstrap.cc:654] T d1471c9f15264b62812a9341b75d33f6 P ad8603b082d240c287b1cf61856b67ce: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:27.272136 6769 tablet_bootstrap.cc:492] T d1471c9f15264b62812a9341b75d33f6 P 41fed7f820404f16aa2d506396ef227c: No bootstrap required, opened a new log
I20260925 05:01:27.272282 6769 ts_tablet_manager.cc:1397] T d1471c9f15264b62812a9341b75d33f6 P 41fed7f820404f16aa2d506396ef227c: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 05:01:27.273185 6769 raft_consensus.cc:348] T d1471c9f15264b62812a9341b75d33f6 P 41fed7f820404f16aa2d506396ef227c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fed7f820404f16aa2d506396ef227c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33901 } } peers { permanent_uuid: "87f2a171c1c8411e93504e0b8a47b426" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 43559 } } peers { permanent_uuid: "ad8603b082d240c287b1cf61856b67ce" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41837 } }
I20260925 05:01:27.273437 6769 raft_consensus.cc:374] T d1471c9f15264b62812a9341b75d33f6 P 41fed7f820404f16aa2d506396ef227c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:27.273522 6769 raft_consensus.cc:729] T d1471c9f15264b62812a9341b75d33f6 P 41fed7f820404f16aa2d506396ef227c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41fed7f820404f16aa2d506396ef227c, State: Initialized, Role: FOLLOWER
I20260925 05:01:27.273756 6769 consensus_queue.cc:260] T d1471c9f15264b62812a9341b75d33f6 P 41fed7f820404f16aa2d506396ef227c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fed7f820404f16aa2d506396ef227c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33901 } } peers { permanent_uuid: "87f2a171c1c8411e93504e0b8a47b426" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 43559 } } peers { permanent_uuid: "ad8603b082d240c287b1cf61856b67ce" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41837 } }
I20260925 05:01:27.274286 6767 tablet_bootstrap.cc:492] T d1471c9f15264b62812a9341b75d33f6 P ad8603b082d240c287b1cf61856b67ce: No bootstrap required, opened a new log
I20260925 05:01:27.274437 6767 ts_tablet_manager.cc:1397] T d1471c9f15264b62812a9341b75d33f6 P ad8603b082d240c287b1cf61856b67ce: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 05:01:27.274575 6769 ts_tablet_manager.cc:1428] T d1471c9f15264b62812a9341b75d33f6 P 41fed7f820404f16aa2d506396ef227c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 05:01:27.275399 6767 raft_consensus.cc:348] T d1471c9f15264b62812a9341b75d33f6 P ad8603b082d240c287b1cf61856b67ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fed7f820404f16aa2d506396ef227c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33901 } } peers { permanent_uuid: "87f2a171c1c8411e93504e0b8a47b426" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 43559 } } peers { permanent_uuid: "ad8603b082d240c287b1cf61856b67ce" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41837 } }
I20260925 05:01:27.275672 6767 raft_consensus.cc:374] T d1471c9f15264b62812a9341b75d33f6 P ad8603b082d240c287b1cf61856b67ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:27.275764 6767 raft_consensus.cc:729] T d1471c9f15264b62812a9341b75d33f6 P ad8603b082d240c287b1cf61856b67ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad8603b082d240c287b1cf61856b67ce, State: Initialized, Role: FOLLOWER
I20260925 05:01:27.276006 6767 consensus_queue.cc:260] T d1471c9f15264b62812a9341b75d33f6 P ad8603b082d240c287b1cf61856b67ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fed7f820404f16aa2d506396ef227c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33901 } } peers { permanent_uuid: "87f2a171c1c8411e93504e0b8a47b426" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 43559 } } peers { permanent_uuid: "ad8603b082d240c287b1cf61856b67ce" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41837 } }
I20260925 05:01:27.276494 6384 heartbeater.cc:503] Master 127.30.247.60:41659 was elected leader, sending a full tablet report...
I20260925 05:01:27.276960 6767 ts_tablet_manager.cc:1428] T d1471c9f15264b62812a9341b75d33f6 P ad8603b082d240c287b1cf61856b67ce: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:01:27.279132 6665 heartbeater.cc:503] Master 127.30.247.60:41659 was elected leader, sending a full tablet report...
I20260925 05:01:27.284193 6773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.274740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.30.247.1:41517)
0925 05:01:27.283661 (+ 8921us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.283852 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8762,"thread_start_us":8614,"threads_started":1}
I20260925 05:01:27.285122 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.277407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.30.247.3:45141)
0925 05:01:27.284811 (+ 7404us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.284928 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7276,"thread_start_us":7153,"threads_started":1}
I20260925 05:01:27.285153 6773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.284830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.30.247.1:33099)
0925 05:01:27.284976 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.285046 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 05:01:27.285897 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.285583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.30.247.3:48909)
0925 05:01:27.285732 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.285806 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 05:01:27.601397 6766 raft_consensus.cc:482] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:01:27.601612 6766 raft_consensus.cc:504] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fed7f820404f16aa2d506396ef227c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33901 } } peers { permanent_uuid: "87f2a171c1c8411e93504e0b8a47b426" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 43559 } } peers { permanent_uuid: "ad8603b082d240c287b1cf61856b67ce" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41837 } }
I20260925 05:01:27.602268 6766 leader_election.cc:288] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41fed7f820404f16aa2d506396ef227c (127.30.247.1:33901), ad8603b082d240c287b1cf61856b67ce (127.30.247.3:41837)
I20260925 05:01:27.602720 6338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1471c9f15264b62812a9341b75d33f6" candidate_uuid: "87f2a171c1c8411e93504e0b8a47b426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41fed7f820404f16aa2d506396ef227c" is_pre_election: true
I20260925 05:01:27.602818 6620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1471c9f15264b62812a9341b75d33f6" candidate_uuid: "87f2a171c1c8411e93504e0b8a47b426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad8603b082d240c287b1cf61856b67ce" is_pre_election: true
I20260925 05:01:27.602980 6338 raft_consensus.cc:2454] T d1471c9f15264b62812a9341b75d33f6 P 41fed7f820404f16aa2d506396ef227c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87f2a171c1c8411e93504e0b8a47b426 in term 0.
I20260925 05:01:27.603118 6620 raft_consensus.cc:2454] T d1471c9f15264b62812a9341b75d33f6 P ad8603b082d240c287b1cf61856b67ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87f2a171c1c8411e93504e0b8a47b426 in term 0.
I20260925 05:01:27.603440 6412 leader_election.cc:302] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41fed7f820404f16aa2d506396ef227c, 87f2a171c1c8411e93504e0b8a47b426; no voters:
I20260925 05:01:27.603701 6766 raft_consensus.cc:2781] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:01:27.603834 6766 raft_consensus.cc:482] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:01:27.603926 6766 raft_consensus.cc:3037] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:27.606024 6766 raft_consensus.cc:504] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fed7f820404f16aa2d506396ef227c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33901 } } peers { permanent_uuid: "87f2a171c1c8411e93504e0b8a47b426" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 43559 } } peers { permanent_uuid: "ad8603b082d240c287b1cf61856b67ce" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41837 } }
I20260925 05:01:27.606561 6766 leader_election.cc:288] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426 [CANDIDATE]: Term 1 election: Requested vote from peers 41fed7f820404f16aa2d506396ef227c (127.30.247.1:33901), ad8603b082d240c287b1cf61856b67ce (127.30.247.3:41837)
I20260925 05:01:27.606897 6338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1471c9f15264b62812a9341b75d33f6" candidate_uuid: "87f2a171c1c8411e93504e0b8a47b426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41fed7f820404f16aa2d506396ef227c"
I20260925 05:01:27.607077 6338 raft_consensus.cc:3037] T d1471c9f15264b62812a9341b75d33f6 P 41fed7f820404f16aa2d506396ef227c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:27.607002 6620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1471c9f15264b62812a9341b75d33f6" candidate_uuid: "87f2a171c1c8411e93504e0b8a47b426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad8603b082d240c287b1cf61856b67ce"
I20260925 05:01:27.607223 6620 raft_consensus.cc:3037] T d1471c9f15264b62812a9341b75d33f6 P ad8603b082d240c287b1cf61856b67ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:27.609099 6338 raft_consensus.cc:2454] T d1471c9f15264b62812a9341b75d33f6 P 41fed7f820404f16aa2d506396ef227c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87f2a171c1c8411e93504e0b8a47b426 in term 1.
I20260925 05:01:27.609177 6620 raft_consensus.cc:2454] T d1471c9f15264b62812a9341b75d33f6 P ad8603b082d240c287b1cf61856b67ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87f2a171c1c8411e93504e0b8a47b426 in term 1.
I20260925 05:01:27.609489 6412 leader_election.cc:302] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41fed7f820404f16aa2d506396ef227c, 87f2a171c1c8411e93504e0b8a47b426; no voters:
I20260925 05:01:27.609762 6766 raft_consensus.cc:2781] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:01:27.610183 6766 raft_consensus.cc:686] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426 [term 1 LEADER]: Becoming Leader. State: Replica: 87f2a171c1c8411e93504e0b8a47b426, State: Running, Role: LEADER
I20260925 05:01:27.610733 6766 consensus_queue.cc:237] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fed7f820404f16aa2d506396ef227c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33901 } } peers { permanent_uuid: "87f2a171c1c8411e93504e0b8a47b426" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 43559 } } peers { permanent_uuid: "ad8603b082d240c287b1cf61856b67ce" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41837 } }
I20260925 05:01:27.611222 6768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.610759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.30.247.2:60345)
0925 05:01:27.610940 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.611047 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 05:01:27.613626 6203 catalog_manager.cc:5495] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87f2a171c1c8411e93504e0b8a47b426 (127.30.247.2). New cstate: current_term: 1 leader_uuid: "87f2a171c1c8411e93504e0b8a47b426" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fed7f820404f16aa2d506396ef227c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87f2a171c1c8411e93504e0b8a47b426" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 43559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad8603b082d240c287b1cf61856b67ce" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41837 } health_report { overall_health: UNKNOWN } } }
I20260925 05:01:27.637272 6763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.636873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.0.0.1:50294)
0925 05:01:27.637059 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.637144 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 05:01:27.638149 6763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.637824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.0.0.1:50308)
0925 05:01:27.637961 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.638045 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 05:01:27.695732 6761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.690964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54704 (local address 127.30.247.2:43559)
0925 05:01:27.691214 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:27.691220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:27.691239 (+ 19us) server_negotiation.cc:407] Connection header received
0925 05:01:27.691429 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:27.691435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:27.691539 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:27.691634 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:01:27.691998 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:27.692839 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:27.694110 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:27.694525 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:27.694660 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:27.695044 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:01:27.695147 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:01:27.695242 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:27.695388 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 05:01:27.695462 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260925 05:01:27.698580 6620 raft_consensus.cc:1261] T d1471c9f15264b62812a9341b75d33f6 P ad8603b082d240c287b1cf61856b67ce [term 1 FOLLOWER]: Refusing update from remote peer 87f2a171c1c8411e93504e0b8a47b426: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:01:27.698714 6338 raft_consensus.cc:1261] T d1471c9f15264b62812a9341b75d33f6 P 41fed7f820404f16aa2d506396ef227c [term 1 FOLLOWER]: Refusing update from remote peer 87f2a171c1c8411e93504e0b8a47b426: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:01:27.699209 6766 consensus_queue.cc:1035] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad8603b082d240c287b1cf61856b67ce" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:01:27.699503 6776 consensus_queue.cc:1035] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41fed7f820404f16aa2d506396ef227c" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 33901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:01:27.703323 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.702637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.30.247.3:46035)
0925 05:01:27.702912 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.703076 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260925 05:01:27.704854 6773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.704285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.30.247.1:36835)
0925 05:01:27.704469 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.704610 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 05:01:27.709475 6768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.708970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.30.247.2:55677)
0925 05:01:27.709181 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.709296 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 05:01:27.710989 6780 mvcc.cc:205] Tried to move back new op lower bound from 7333119949607731200 to 7333119949256122368. Current Snapshot: MvccSnapshot[applied={T|T < 7333119949607731200 or (T in {7333119949607731200})}]
I20260925 05:01:27.715268 6203 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:01:27.715612 6203 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:01:27.719288 6773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.717089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.30.247.1:44417)
0925 05:01:27.719024 (+ 1935us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.719123 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1822}
I20260925 05:01:27.721086 6768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.720654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.30.247.2:39247)
0925 05:01:27.720854 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.720935 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260925 05:01:27.723053 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.722472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.30.247.3:39677)
0925 05:01:27.722805 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:27.722887 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221}
I20260925 05:01:27.723641 6203 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b27f6d41f6f1426bb70abaf6985e00b2: Sending DeleteTablet for 3 replicas of tablet d1471c9f15264b62812a9341b75d33f6
I20260925 05:01:27.724776 6319 tablet_service.cc:1514] Processing DeleteTablet for tablet d1471c9f15264b62812a9341b75d33f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:01:27 UTC) from {username='oryx', principal='oryx/127.30.247.60@KRBTEST.COM'} at 127.0.0.1:51156
I20260925 05:01:27.724864 6600 tablet_service.cc:1514] Processing DeleteTablet for tablet d1471c9f15264b62812a9341b75d33f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:01:27 UTC) from {username='oryx', principal='oryx/127.30.247.60@KRBTEST.COM'} at 127.0.0.1:60762
I20260925 05:01:27.724907 6599 tablet_service.cc:1514] Processing DeleteTablet for tablet d1471c9f15264b62812a9341b75d33f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:01:27 UTC) from {username='oryx', principal='oryx/127.30.247.60@KRBTEST.COM'} at 127.0.0.1:60762
I20260925 05:01:27.725402 6459 tablet_service.cc:1514] Processing DeleteTablet for tablet d1471c9f15264b62812a9341b75d33f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:01:27 UTC) from {username='oryx', principal='oryx/127.30.247.60@KRBTEST.COM'} at 127.0.0.1:54700
I20260925 05:01:27.726058 6789 tablet_replica.cc:330] T d1471c9f15264b62812a9341b75d33f6 P 41fed7f820404f16aa2d506396ef227c: stopping tablet replica
I20260925 05:01:27.726243 6789 raft_consensus.cc:2229] T d1471c9f15264b62812a9341b75d33f6 P 41fed7f820404f16aa2d506396ef227c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:01:27.726483 6789 raft_consensus.cc:2258] T d1471c9f15264b62812a9341b75d33f6 P 41fed7f820404f16aa2d506396ef227c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:01:27.727578 6791 tablet_replica.cc:330] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426: stopping tablet replica
I20260925 05:01:27.727613 6789 ts_tablet_manager.cc:1905] T d1471c9f15264b62812a9341b75d33f6 P 41fed7f820404f16aa2d506396ef227c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:01:27.727756 6791 raft_consensus.cc:2229] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:01:27.728082 6791 raft_consensus.cc:2258] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:01:27.728587 6790 tablet_replica.cc:330] T d1471c9f15264b62812a9341b75d33f6 P ad8603b082d240c287b1cf61856b67ce: stopping tablet replica
I20260925 05:01:27.728749 6790 raft_consensus.cc:2229] T d1471c9f15264b62812a9341b75d33f6 P ad8603b082d240c287b1cf61856b67ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:01:27.728995 6790 raft_consensus.cc:2258] T d1471c9f15264b62812a9341b75d33f6 P ad8603b082d240c287b1cf61856b67ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:01:27.729666 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 6176
I20260925 05:01:27.730031 6791 ts_tablet_manager.cc:1905] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:01:27.730064 6790 ts_tablet_manager.cc:1905] T d1471c9f15264b62812a9341b75d33f6 P ad8603b082d240c287b1cf61856b67ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:01:27.733989 6789 ts_tablet_manager.cc:1918] T d1471c9f15264b62812a9341b75d33f6 P 41fed7f820404f16aa2d506396ef227c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:01:27.734115 6789 log.cc:1196] T d1471c9f15264b62812a9341b75d33f6 P 41fed7f820404f16aa2d506396ef227c: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-0/wal/wals/d1471c9f15264b62812a9341b75d33f6
I20260925 05:01:27.734560 6789 ts_tablet_manager.cc:1939] T d1471c9f15264b62812a9341b75d33f6 P 41fed7f820404f16aa2d506396ef227c: Deleting consensus metadata
I20260925 05:01:27.735481 6188 catalog_manager.cc:4841] TS 41fed7f820404f16aa2d506396ef227c (127.30.247.1:33901): tablet d1471c9f15264b62812a9341b75d33f6 (table test-table [id=095fac869fd4434699a612628d4e7b54]) successfully deleted
I20260925 05:01:27.736109 6791 ts_tablet_manager.cc:1918] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:01:27.736254 6791 log.cc:1196] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-1/wal/wals/d1471c9f15264b62812a9341b75d33f6
I20260925 05:01:27.736683 6791 ts_tablet_manager.cc:1939] T d1471c9f15264b62812a9341b75d33f6 P 87f2a171c1c8411e93504e0b8a47b426: Deleting consensus metadata
I20260925 05:01:27.737658 6188 catalog_manager.cc:4841] TS 87f2a171c1c8411e93504e0b8a47b426 (127.30.247.2:43559): tablet d1471c9f15264b62812a9341b75d33f6 (table test-table [id=095fac869fd4434699a612628d4e7b54]) successfully deleted
I20260925 05:01:27.738067 6790 ts_tablet_manager.cc:1918] T d1471c9f15264b62812a9341b75d33f6 P ad8603b082d240c287b1cf61856b67ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:01:27.738159 6790 log.cc:1196] T d1471c9f15264b62812a9341b75d33f6 P ad8603b082d240c287b1cf61856b67ce: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312430481417-31708-0/minicluster-data/ts-2/wal/wals/d1471c9f15264b62812a9341b75d33f6
I20260925 05:01:27.738551 6790 ts_tablet_manager.cc:1939] T d1471c9f15264b62812a9341b75d33f6 P ad8603b082d240c287b1cf61856b67ce: Deleting consensus metadata
I20260925 05:01:27.739485 6189 catalog_manager.cc:4841] TS ad8603b082d240c287b1cf61856b67ce (127.30.247.3:41837): tablet d1471c9f15264b62812a9341b75d33f6 (table test-table [id=095fac869fd4434699a612628d4e7b54]) successfully deleted
W20260925 05:01:27.739841 6189 catalog_manager.cc:4808] TS ad8603b082d240c287b1cf61856b67ce (127.30.247.3:41837): delete failed for tablet d1471c9f15264b62812a9341b75d33f6 because the tablet was not found. No further retry: Not found: Tablet not found: d1471c9f15264b62812a9341b75d33f6
I20260925 05:01:27.754616 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.730556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51158 (local address 127.30.247.1:33901)
0925 05:01:27.730740 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:27.730747 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:27.737095 (+ 6348us) server_negotiation.cc:407] Connection header received
0925 05:01:27.737193 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:27.737198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:27.737267 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:27.737361 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:27.738817 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:27.739774 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:27.741078 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:27.741480 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:27.741604 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:27.749375 (+ 7771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:27.749389 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:27.749406 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:27.749439 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:27.751919 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:27.752658 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:27.752665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:27.752667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:27.752796 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:27.753162 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:27.753168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:27.753176 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:27.753480 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:27.753784 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:27.754072 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 05:01:27.754290 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 05:01:27.755234 6379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:27.848150 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 6176
I20260925 05:01:27.866182 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 6396
I20260925 05:01:27.879750 6761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:27.866946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54712 (local address 127.30.247.2:43559)
0925 05:01:27.867131 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:27.867137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:27.867278 (+ 141us) server_negotiation.cc:407] Connection header received
0925 05:01:27.867490 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:27.867496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:27.867581 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:27.867690 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:27.869136 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:27.869854 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:27.871055 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:27.871456 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:27.871590 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:27.874156 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:27.874179 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:27.874195 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:27.874226 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:27.877164 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:27.877793 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:27.877802 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:27.877806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:27.877942 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:27.878295 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:27.878301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:27.878304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:27.878641 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:27.878935 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:27.879234 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 05:01:27.879461 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 05:01:27.880429 6519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:28.001658 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 6396
I20260925 05:01:28.023236 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 6537
I20260925 05:01:28.036317 6795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:28.023994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60776 (local address 127.30.247.3:41837)
0925 05:01:28.024420 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:28.024425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:28.024441 (+ 16us) server_negotiation.cc:407] Connection header received
0925 05:01:28.024494 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:28.024497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:28.024622 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:28.024721 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:28.026003 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:28.026692 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:28.027886 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:28.028255 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:28.028347 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:28.031119 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:28.031133 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:28.031148 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:28.031183 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:28.033854 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:28.034451 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:28.034458 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:28.034460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:28.034571 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:28.034904 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:28.034906 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:28.034908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:28.035324 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:28.035581 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:28.035809 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 05:01:28.035958 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":220,"threads_started":1}
I20260925 05:01:28.037074 6660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:28.129091 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 6537
I20260925 05:01:28.146667 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 5873
I20260925 05:01:28.147683 6040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:28.248392 6798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:28.247623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.0.0.1:50324)
0925 05:01:28.248123 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:28.248217 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":272,"threads_started":1}
I20260925 05:01:28.257788 6798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:28.257351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:44265 (local address 127.0.0.1:50336)
0925 05:01:28.257537 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:28.257643 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 05:01:28.265311 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 5873
I20260925 05:01:28.285856 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 5968
I20260925 05:01:28.286988 6234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:28.405689 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 5968
2026-09-25T05:01:28Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9665 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:01:28 dist-test-slave-5n86 krb5kdc[6803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:01:28 dist-test-slave-5n86 krb5kdc[6803](info): set up 2 sockets
Sep 25 05:01:28 dist-test-slave-5n86 krb5kdc[6803](info): commencing operation
krb5kdc: starting...
W20260925 05:01:29.497658 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:01:29 dist-test-slave-5n86 krb5kdc[6803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:01:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:01:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.62.keytab.
Entry for principal oryx/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.62.keytab.
I20260925 05:01:29.700217 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:43329
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.62.keytab
--principal=oryx/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:43329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:30.014487 6816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:30.014801 6816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:30.014891 6816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:30.021946 6816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:01:30.022054 6816 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 05:01:30.022095 6816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:30.022156 6816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:30.022205 6816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:01:30.022259 6816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:01:30.031950 6816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:43329
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.62.keytab
--principal=oryx/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:43329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:30.033198 6816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:30.034415 6816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:30.046020 6824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:30.046085 6822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:30.046108 6821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:30.047066 6816 server_base.cc:1034] running on GCE node
I20260925 05:01:30.047861 6816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:30.049289 6816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:30.050506 6816 hybrid_clock.cc:648] HybridClock initialized: now 1790312490050421 us; error 64 us; skew 500 ppm
I20260925 05:01:30.050849 6816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:30 dist-test-slave-5n86 krb5kdc[6803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312490, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:30.053915 6816 init.cc:377] Logged in from keytab as oryx/127.30.247.62@KRBTEST.COM (short username oryx)
I20260925 05:01:30.055305 6816 webserver.cc:460] Webserver started at http://127.30.247.62:34787/ using document root <none> and password file <none>
I20260925 05:01:30.055948 6816 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:30.056032 6816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:30.056293 6816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:30.059125 6816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "32cd628ad4834acdba076e9b198ed8dc"
format_stamp: "Formatted at 2026-09-25 05:01:30 on dist-test-slave-5n86"
server_key: "d2c99148c8464e202d1db3cf729e0eef"
server_key_iv: "7edbc052d89a6cc4f20166888c575e91"
server_key_version: "encryptionkey@0"
I20260925 05:01:30.059842 6816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "32cd628ad4834acdba076e9b198ed8dc"
format_stamp: "Formatted at 2026-09-25 05:01:30 on dist-test-slave-5n86"
server_key: "d2c99148c8464e202d1db3cf729e0eef"
server_key_iv: "7edbc052d89a6cc4f20166888c575e91"
server_key_version: "encryptionkey@0"
I20260925 05:01:30.064805 6816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 05:01:30.068107 6831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:30.069540 6816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:01:30.069733 6816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "32cd628ad4834acdba076e9b198ed8dc"
format_stamp: "Formatted at 2026-09-25 05:01:30 on dist-test-slave-5n86"
server_key: "d2c99148c8464e202d1db3cf729e0eef"
server_key_iv: "7edbc052d89a6cc4f20166888c575e91"
server_key_version: "encryptionkey@0"
I20260925 05:01:30.069957 6816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:30.083534 6816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:30.087469 6816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:30.087746 6816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:30.109133 6816 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:43329
I20260925 05:01:30.109157 6892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:43329 every 8 connection(s)
I20260925 05:01:30.110512 6816 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:01:30.114183 6893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:30.118335 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 6816
I20260925 05:01:30.118537 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:01:30.119349 31708 external_mini_cluster.cc:1428] Setting key f8e3bb62e26c640a073799e558b424c5
I20260925 05:01:30.125682 6893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32cd628ad4834acdba076e9b198ed8dc: Bootstrap starting.
I20260925 05:01:30.129175 6893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32cd628ad4834acdba076e9b198ed8dc: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:30.130285 6893 log.cc:824] T 00000000000000000000000000000000 P 32cd628ad4834acdba076e9b198ed8dc: Log is configured to *not* fsync() on all Append() calls
I20260925 05:01:30.133435 6893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32cd628ad4834acdba076e9b198ed8dc: No bootstrap required, opened a new log
Sep 25 05:01:30 dist-test-slave-5n86 krb5kdc[6803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM
I20260925 05:01:30.143971 6896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:30.121397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51972 (local address 127.30.247.62:43329)
0925 05:01:30.122137 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:30.122148 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:30.122188 (+ 40us) server_negotiation.cc:407] Connection header received
0925 05:01:30.124155 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:30.124177 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:30.124600 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:30.124908 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:30.126567 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:30.127831 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:30.129624 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:30.130316 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:30.130518 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:30.135209 (+ 4691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:30.135219 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:30.135237 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:30.135296 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:30.139358 (+ 4062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:30.140075 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:30.140084 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:30.140093 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:30.140293 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:30.140717 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:30.140722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:30.140726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:30.141437 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:30.141743 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:30.142096 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 05:01:30.142566 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":330,"threads_started":1}
I20260925 05:01:30.144212 6893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32cd628ad4834acdba076e9b198ed8dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cd628ad4834acdba076e9b198ed8dc" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 43329 } }
I20260925 05:01:30.144637 6893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32cd628ad4834acdba076e9b198ed8dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:30.144731 6893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32cd628ad4834acdba076e9b198ed8dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32cd628ad4834acdba076e9b198ed8dc, State: Initialized, Role: FOLLOWER
I20260925 05:01:30.145316 6893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32cd628ad4834acdba076e9b198ed8dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cd628ad4834acdba076e9b198ed8dc" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 43329 } }
I20260925 05:01:30.145570 6893 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32cd628ad4834acdba076e9b198ed8dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:01:30.145725 6893 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32cd628ad4834acdba076e9b198ed8dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:01:30.145915 6893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32cd628ad4834acdba076e9b198ed8dc [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:30.148164 6893 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32cd628ad4834acdba076e9b198ed8dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cd628ad4834acdba076e9b198ed8dc" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 43329 } }
I20260925 05:01:30.148705 6893 leader_election.cc:302] T 00000000000000000000000000000000 P 32cd628ad4834acdba076e9b198ed8dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32cd628ad4834acdba076e9b198ed8dc; no voters:
I20260925 05:01:30.149278 6893 leader_election.cc:288] T 00000000000000000000000000000000 P 32cd628ad4834acdba076e9b198ed8dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:01:30.149420 6898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32cd628ad4834acdba076e9b198ed8dc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:01:30.150807 6898 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32cd628ad4834acdba076e9b198ed8dc [term 1 LEADER]: Becoming Leader. State: Replica: 32cd628ad4834acdba076e9b198ed8dc, State: Running, Role: LEADER
I20260925 05:01:30.151309 6898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32cd628ad4834acdba076e9b198ed8dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cd628ad4834acdba076e9b198ed8dc" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 43329 } }
I20260925 05:01:30.151477 6893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32cd628ad4834acdba076e9b198ed8dc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:01:30.154093 6899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32cd628ad4834acdba076e9b198ed8dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32cd628ad4834acdba076e9b198ed8dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cd628ad4834acdba076e9b198ed8dc" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 43329 } } }
I20260925 05:01:30.154495 6899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32cd628ad4834acdba076e9b198ed8dc [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:30.154958 6900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32cd628ad4834acdba076e9b198ed8dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32cd628ad4834acdba076e9b198ed8dc. Latest consensus state: current_term: 1 leader_uuid: "32cd628ad4834acdba076e9b198ed8dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cd628ad4834acdba076e9b198ed8dc" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 43329 } } }
I20260925 05:01:30.155216 6900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32cd628ad4834acdba076e9b198ed8dc [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:30.156769 6907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:01:30.161579 6907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:01:30.170655 6907 catalog_manager.cc:1347] Generated new cluster ID: 46e4276b16574a70b81573965dfa0025
I20260925 05:01:30.170773 6907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:01:30.202687 6907 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:01:30.204047 6907 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:01:30.211556 6907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32cd628ad4834acdba076e9b198ed8dc: Generated new TSK 0
I20260925 05:01:30.212337 6907 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.30.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.1.keytab.
Entry for principal oryx/127.30.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.1.keytab.
I20260925 05:01:30.277132 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:0
--local_ip_for_outbound_sockets=127.30.247.1
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:43329
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.1.keytab
--principal=oryx/127.30.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:30.599088 6919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:30.599342 6919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:30.599506 6919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:30.606462 6919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:30.606554 6919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:30.606721 6919 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:01:30.606858 6919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:01:30.615921 6919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.1.keytab
--principal=oryx/127.30.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.247.62:43329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:30.617079 6919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:30.618803 6919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:30.631357 6924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:30.631409 6927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:30.631367 6925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:30.632223 6919 server_base.cc:1034] running on GCE node
I20260925 05:01:30.632890 6919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:30.633765 6919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:30.635020 6919 hybrid_clock.cc:648] HybridClock initialized: now 1790312490634955 us; error 41 us; skew 500 ppm
I20260925 05:01:30.635324 6919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:30 dist-test-slave-5n86 krb5kdc[6803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312490, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:30.638343 6919 init.cc:377] Logged in from keytab as oryx/127.30.247.1@KRBTEST.COM (short username oryx)
I20260925 05:01:30.639739 6919 webserver.cc:460] Webserver started at http://127.30.247.1:37589/ using document root <none> and password file <none>
I20260925 05:01:30.640403 6919 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:30.640573 6919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:30.640913 6919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:30.643846 6919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-0/data/instance:
uuid: "a78bbd0e639a4d2cb96c920d629cf399"
format_stamp: "Formatted at 2026-09-25 05:01:30 on dist-test-slave-5n86"
server_key: "376f4f91b6698398f96af3e614762e3d"
server_key_iv: "b69e3a6a524ac676d8431a0c7abb0357"
server_key_version: "encryptionkey@0"
I20260925 05:01:30.644735 6919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance:
uuid: "a78bbd0e639a4d2cb96c920d629cf399"
format_stamp: "Formatted at 2026-09-25 05:01:30 on dist-test-slave-5n86"
server_key: "376f4f91b6698398f96af3e614762e3d"
server_key_iv: "b69e3a6a524ac676d8431a0c7abb0357"
server_key_version: "encryptionkey@0"
I20260925 05:01:30.649876 6919 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 05:01:30.653592 6934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:30.654924 6919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 05:01:30.655092 6919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "a78bbd0e639a4d2cb96c920d629cf399"
format_stamp: "Formatted at 2026-09-25 05:01:30 on dist-test-slave-5n86"
server_key: "376f4f91b6698398f96af3e614762e3d"
server_key_iv: "b69e3a6a524ac676d8431a0c7abb0357"
server_key_version: "encryptionkey@0"
I20260925 05:01:30.655303 6919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:30.703465 6919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:30.707551 6919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:30.707823 6919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:30.710464 6919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:30.710577 6919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:30.710757 6919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:30.710829 6919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 05:01:30 dist-test-slave-5n86 krb5kdc[6803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312490, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM
I20260925 05:01:30.735471 6943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:30.710993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:37309 (local address 127.30.247.62:43329)
0925 05:01:30.714091 (+ 3098us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:30.714096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:30.714114 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:01:30.716298 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:30.716304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:30.716402 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:30.716503 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:30.718961 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:30.719930 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:30.723380 (+ 3450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:30.723800 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:30.723949 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:30.728397 (+ 4448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:30.728406 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:30.728413 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:30.728458 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:30.732121 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:30.733192 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:30.733200 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:30.733203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:30.733334 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:30.733774 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:30.733780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:30.733783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:30.734051 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:30.734204 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:30.734887 (+ 683us) server_negotiation.cc:299] Negotiation successful
0925 05:01:30.735094 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2996,"thread_start_us":2908,"threads_started":1}
I20260925 05:01:30.736435 6944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:30.711763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:43329 (local address 127.30.247.1:37309)
0925 05:01:30.712848 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:30.712908 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:30.715939 (+ 3031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:30.716831 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:30.716846 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:30.717672 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:30.718743 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:30.718773 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:30.720143 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:30.720153 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:30.723197 (+ 3044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:30.723217 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:30.724041 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:30.724050 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:30.724270 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:30.725000 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:30.725021 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:30.728102 (+ 3081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:30.732285 (+ 4183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:30.732300 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:30.732318 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:30.733012 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:30.733457 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:30.733462 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:30.733466 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:30.733654 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:30.734180 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:30.734188 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:30.734535 (+ 347us) client_negotiation.cc:769] Sending connection context
0925 05:01:30.734793 (+ 258us) client_negotiation.cc:240] Negotiation successful
0925 05:01:30.735295 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":414,"threads_started":1}
I20260925 05:01:30.757246 6943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:30.742718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:50443 (local address 127.30.247.62:43329)
0925 05:01:30.742937 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:30.742945 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:30.743442 (+ 497us) server_negotiation.cc:407] Connection header received
0925 05:01:30.743584 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:30.743592 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:30.743693 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:30.743811 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:30.744743 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:30.745777 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:30.747494 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:30.747915 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:30.748041 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:30.750494 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:30.750503 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:30.750510 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:30.750568 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:30.754326 (+ 3758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:30.755128 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:30.755136 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:30.755143 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:30.755331 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:30.755795 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:30.755801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:30.755804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:30.756072 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:30.756226 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:30.756638 (+ 412us) server_negotiation.cc:299] Negotiation successful
0925 05:01:30.756855 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 05:01:30.757366 7006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:30.742712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:43329 (local address 127.30.247.1:50443)
0925 05:01:30.743330 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:30.743358 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:30.743465 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:30.743842 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:30.743845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:30.744279 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:30.744596 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:30.744617 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:30.745926 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:30.745935 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:30.747356 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:30.747368 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:30.748102 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:30.748109 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:30.748264 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:30.748968 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:30.749005 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:30.750294 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:30.754449 (+ 4155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:30.754453 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:30.754456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:30.754986 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:30.755452 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:30.755457 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:30.755472 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:30.755614 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:30.756188 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:30.756193 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:30.756366 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 05:01:30.756517 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 05:01:30.756890 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":340,"threads_started":1}
I20260925 05:01:30.766722 6919 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:36451
I20260925 05:01:30.766744 7054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:36451 every 8 connection(s)
I20260925 05:01:30.767957 6919 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:01:30.769974 7055 heartbeater.cc:347] Connected to a master server at 127.30.247.62:43329
I20260925 05:01:30.770174 7055 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:30.770721 7055 heartbeater.cc:511] Master 127.30.247.62:43329 requested a full tablet report, sending...
I20260925 05:01:30.772639 6848 ts_manager.cc:194] Registered new tserver with Master: a78bbd0e639a4d2cb96c920d629cf399 (127.30.247.1:36451)
I20260925 05:01:30.774170 6848 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.247.1@KRBTEST.COM'} at 127.30.247.1:37309
I20260925 05:01:30.777841 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 6919
I20260925 05:01:30.777990 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance
I20260925 05:01:30.778551 31708 external_mini_cluster.cc:1428] Setting key 1d4565bb9c43a9b2d340d9cc3e5c0417
WARNING: no policy specified for oryx/127.30.247.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.2.keytab.
Entry for principal oryx/127.30.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.2.keytab.
I20260925 05:01:30.823366 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:0
--local_ip_for_outbound_sockets=127.30.247.2
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:43329
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.2.keytab
--principal=oryx/127.30.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:31.137343 7059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:31.137639 7059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:31.137801 7059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:31.145103 7059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:31.145246 7059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:31.145395 7059 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:01:31.145538 7059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:01:31.155532 7059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.2.keytab
--principal=oryx/127.30.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.247.62:43329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:31.156874 7059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:31.158174 7059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:31.171123 7065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:31.171393 7064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:31.171610 7067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:31.171950 7059 server_base.cc:1034] running on GCE node
I20260925 05:01:31.172525 7059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:31.173547 7059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:31.174790 7059 hybrid_clock.cc:648] HybridClock initialized: now 1790312491174729 us; error 45 us; skew 500 ppm
I20260925 05:01:31.175091 7059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:31 dist-test-slave-5n86 krb5kdc[6803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312491, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:31.178320 7059 init.cc:377] Logged in from keytab as oryx/127.30.247.2@KRBTEST.COM (short username oryx)
I20260925 05:01:31.179801 7059 webserver.cc:460] Webserver started at http://127.30.247.2:33089/ using document root <none> and password file <none>
I20260925 05:01:31.180521 7059 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:31.180703 7059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:31.181025 7059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:31.183777 7059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-1/data/instance:
uuid: "892c09a35d4248359563299977a06214"
format_stamp: "Formatted at 2026-09-25 05:01:31 on dist-test-slave-5n86"
server_key: "439d824d458df48e820ac563247bff0e"
server_key_iv: "086f61af382d2d2562505bccbb5d8373"
server_key_version: "encryptionkey@0"
I20260925 05:01:31.184618 7059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance:
uuid: "892c09a35d4248359563299977a06214"
format_stamp: "Formatted at 2026-09-25 05:01:31 on dist-test-slave-5n86"
server_key: "439d824d458df48e820ac563247bff0e"
server_key_iv: "086f61af382d2d2562505bccbb5d8373"
server_key_version: "encryptionkey@0"
I20260925 05:01:31.189391 7059 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260925 05:01:31.192757 7074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:31.193974 7059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 05:01:31.194145 7059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "892c09a35d4248359563299977a06214"
format_stamp: "Formatted at 2026-09-25 05:01:31 on dist-test-slave-5n86"
server_key: "439d824d458df48e820ac563247bff0e"
server_key_iv: "086f61af382d2d2562505bccbb5d8373"
server_key_version: "encryptionkey@0"
I20260925 05:01:31.194356 7059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:31.208015 7059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:31.212244 7059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:31.212519 7059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:31.215013 7059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:31.215134 7059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:31.215306 7059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:31.215390 7059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 05:01:31 dist-test-slave-5n86 krb5kdc[6803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312491, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM
I20260925 05:01:31.240365 6943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:31.215368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:39289 (local address 127.30.247.62:43329)
0925 05:01:31.215574 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:31.215580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:31.216932 (+ 1352us) server_negotiation.cc:407] Connection header received
0925 05:01:31.218935 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:31.218941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:31.219040 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:31.219167 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:31.221804 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:31.223168 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:31.225774 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:31.226420 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:31.226579 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:31.233070 (+ 6491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:31.233079 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:31.233087 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:31.233138 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:31.236744 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:31.237917 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:31.237923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:31.237929 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:31.238063 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:31.238574 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:31.238579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:31.238582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:31.238937 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:31.239101 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:31.239801 (+ 700us) server_negotiation.cc:299] Negotiation successful
0925 05:01:31.240017 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 05:01:31.241411 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:31.215889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:43329 (local address 127.30.247.2:39289)
0925 05:01:31.216761 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:31.216808 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:31.218639 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:31.219606 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:31.219625 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:31.220459 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:31.221557 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:31.221581 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:31.223382 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:31.223389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:31.225548 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:31.225560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:31.226704 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:31.226713 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:31.227084 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:31.227869 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:31.227907 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:31.232840 (+ 4933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:31.236924 (+ 4084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:31.236940 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:31.236959 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:31.237763 (+ 804us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:31.238186 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:31.238194 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:31.238198 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:31.238436 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:31.239071 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:31.239079 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:31.239444 (+ 365us) client_negotiation.cc:769] Sending connection context
0925 05:01:31.239711 (+ 267us) client_negotiation.cc:240] Negotiation successful
0925 05:01:31.240206 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":365,"threads_started":1}
I20260925 05:01:31.266270 7155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:31.251678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:43329 (local address 127.30.247.2:38573)
0925 05:01:31.252469 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:31.252487 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:31.252737 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:31.253091 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:31.253095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:31.253606 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:31.253884 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:31.253893 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:31.254970 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:31.254977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:31.255976 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:31.255995 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:31.256810 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:31.256816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:31.257123 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:31.257837 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:31.257855 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:31.258860 (+ 1005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:31.262605 (+ 3745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:31.262615 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:31.262619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:31.263199 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:31.263660 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:31.263671 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:31.263676 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:31.263856 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:31.264745 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:31.264752 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:31.265000 (+ 248us) client_negotiation.cc:769] Sending connection context
0925 05:01:31.265213 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 05:01:31.265634 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":334,"threads_started":1}
I20260925 05:01:31.267310 6943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:31.252150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:38573 (local address 127.30.247.62:43329)
0925 05:01:31.252308 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:31.252312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:31.252595 (+ 283us) server_negotiation.cc:407] Connection header received
0925 05:01:31.252896 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:31.252902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:31.252966 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:31.253070 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:31.254026 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:31.254852 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:31.256129 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:31.256624 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:31.256759 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:31.259065 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:31.259075 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:31.259082 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:31.259126 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:31.262470 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:31.263309 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:31.263316 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:31.263320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:31.263512 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:31.264012 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:31.264019 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:31.264025 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:31.264496 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:31.264676 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:31.266763 (+ 2087us) server_negotiation.cc:299] Negotiation successful
0925 05:01:31.266922 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 05:01:31.271180 7059 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:41993
I20260925 05:01:31.271241 7193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:41993 every 8 connection(s)
I20260925 05:01:31.272359 7059 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:01:31.274544 7194 heartbeater.cc:347] Connected to a master server at 127.30.247.62:43329
I20260925 05:01:31.274716 7194 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:31.275288 7194 heartbeater.cc:511] Master 127.30.247.62:43329 requested a full tablet report, sending...
I20260925 05:01:31.276469 6848 ts_manager.cc:194] Registered new tserver with Master: 892c09a35d4248359563299977a06214 (127.30.247.2:41993)
I20260925 05:01:31.277283 6848 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.247.2@KRBTEST.COM'} at 127.30.247.2:39289
I20260925 05:01:31.281744 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 7059
I20260925 05:01:31.281898 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance
I20260925 05:01:31.282348 31708 external_mini_cluster.cc:1428] Setting key 69b7a8676fa7dea4a820ef490e51d524
WARNING: no policy specified for oryx/127.30.247.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.3.keytab.
Entry for principal oryx/127.30.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.3.keytab.
I20260925 05:01:31.327960 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:0
--local_ip_for_outbound_sockets=127.30.247.3
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:43329
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.3.keytab
--principal=oryx/127.30.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:31.636775 7198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:31.637085 7198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:31.637255 7198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:31.644557 7198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:31.644685 7198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:31.644835 7198 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:01:31.644973 7198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:01:31.654673 7198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.3.keytab
--principal=oryx/127.30.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.247.62:43329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:31.655925 7198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:31.657295 7198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:31.670053 7204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:31.670248 7206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:31.670369 7203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:31.671190 7198 server_base.cc:1034] running on GCE node
I20260925 05:01:31.671810 7198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:31.672792 7198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:31.674072 7198 hybrid_clock.cc:648] HybridClock initialized: now 1790312491673981 us; error 40 us; skew 500 ppm
I20260925 05:01:31.674382 7198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:31 dist-test-slave-5n86 krb5kdc[6803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312491, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:31.681428 7198 init.cc:377] Logged in from keytab as oryx/127.30.247.3@KRBTEST.COM (short username oryx)
I20260925 05:01:31.682735 7198 webserver.cc:460] Webserver started at http://127.30.247.3:36653/ using document root <none> and password file <none>
I20260925 05:01:31.683331 7198 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:31.683414 7198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:31.683666 7198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:31.686353 7198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-2/data/instance:
uuid: "d8aa4ea6384a4298a04cb4b471fcad4a"
format_stamp: "Formatted at 2026-09-25 05:01:31 on dist-test-slave-5n86"
server_key: "b6c296d1e6ea950447f1413951f55696"
server_key_iv: "def1013d625ca0719e280b52f27a636e"
server_key_version: "encryptionkey@0"
I20260925 05:01:31.687057 7198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance:
uuid: "d8aa4ea6384a4298a04cb4b471fcad4a"
format_stamp: "Formatted at 2026-09-25 05:01:31 on dist-test-slave-5n86"
server_key: "b6c296d1e6ea950447f1413951f55696"
server_key_iv: "def1013d625ca0719e280b52f27a636e"
server_key_version: "encryptionkey@0"
I20260925 05:01:31.691836 7198 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 05:01:31.695099 7213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:31.696391 7198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:01:31.696627 7198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "d8aa4ea6384a4298a04cb4b471fcad4a"
format_stamp: "Formatted at 2026-09-25 05:01:31 on dist-test-slave-5n86"
server_key: "b6c296d1e6ea950447f1413951f55696"
server_key_iv: "def1013d625ca0719e280b52f27a636e"
server_key_version: "encryptionkey@0"
I20260925 05:01:31.696841 7198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:31.715597 7198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:31.719229 7198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:31.719487 7198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:31.722030 7198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:31.722144 7198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:31.722287 7198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:31.722357 7198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 05:01:31 dist-test-slave-5n86 krb5kdc[6803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312491, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM
I20260925 05:01:31.746712 6943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:31.722422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:60915 (local address 127.30.247.62:43329)
0925 05:01:31.722652 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:31.722661 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:31.723948 (+ 1287us) server_negotiation.cc:407] Connection header received
0925 05:01:31.726002 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:31.726016 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:31.726105 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:31.726214 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:31.728290 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:31.729336 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:31.731822 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:31.732352 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:31.732492 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:31.738014 (+ 5522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:31.738023 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:31.738028 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:31.738069 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:31.742258 (+ 4189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:31.743625 (+ 1367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:31.743633 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:31.743639 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:31.743965 (+ 326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:31.744519 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:31.744527 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:31.744556 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:31.744993 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:31.745363 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:31.746120 (+ 757us) server_negotiation.cc:299] Negotiation successful
0925 05:01:31.746341 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:01:31.748116 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:31.723038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:43329 (local address 127.30.247.3:60915)
0925 05:01:31.723802 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:31.723837 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:31.725719 (+ 1882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:31.726459 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:31.726476 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:31.727243 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:31.728110 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:31.728129 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:31.729529 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:31.729533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:31.731605 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:31.731624 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:31.732617 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:31.732626 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:31.732982 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:31.733804 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:31.733850 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:31.737734 (+ 3884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:31.742627 (+ 4893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:31.742641 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:31.742662 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:31.743425 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:31.744118 (+ 693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:31.744136 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:31.744143 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:31.744396 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:31.745183 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:31.745195 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:31.745696 (+ 501us) client_negotiation.cc:769] Sending connection context
0925 05:01:31.746032 (+ 336us) client_negotiation.cc:240] Negotiation successful
0925 05:01:31.746508 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":363,"threads_started":1}
I20260925 05:01:31.776679 7055 heartbeater.cc:503] Master 127.30.247.62:43329 was elected leader, sending a full tablet report...
I20260925 05:01:31.776957 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:31.760422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:43329 (local address 127.30.247.3:40575)
0925 05:01:31.761375 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:31.761403 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:31.761548 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:31.762206 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:31.762215 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:31.762992 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:31.763632 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:31.763652 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:31.765381 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:31.765389 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:31.767263 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:31.767274 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:31.767988 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:31.767994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:31.768166 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:31.768832 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:31.768853 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:31.769625 (+ 772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:31.773608 (+ 3983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:31.773616 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:31.773620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:31.774119 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:31.774650 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:31.774658 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:31.774666 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:31.774837 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:31.775628 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:31.775634 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:31.775870 (+ 236us) client_negotiation.cc:769] Sending connection context
0925 05:01:31.776042 (+ 172us) client_negotiation.cc:240] Negotiation successful
0925 05:01:31.776431 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":390,"threads_started":1}
I20260925 05:01:31.777812 6943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:31.760398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:40575 (local address 127.30.247.62:43329)
0925 05:01:31.760616 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:31.760623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:31.761479 (+ 856us) server_negotiation.cc:407] Connection header received
0925 05:01:31.761735 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:31.761741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:31.761840 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:31.761942 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:31.763855 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:31.765214 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:31.767408 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:31.767809 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:31.767947 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:31.769806 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:31.769814 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:31.769821 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:31.769856 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:31.773448 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:31.774283 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:31.774288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:31.774294 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:31.774450 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:31.774999 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:31.775006 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:31.775008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:31.775390 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:31.775556 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:31.777247 (+ 1691us) server_negotiation.cc:299] Negotiation successful
0925 05:01:31.777453 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 05:01:31.792016 7198 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:34259
I20260925 05:01:31.792037 7332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:34259 every 8 connection(s)
I20260925 05:01:31.793201 7198 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:01:31.795363 7333 heartbeater.cc:347] Connected to a master server at 127.30.247.62:43329
I20260925 05:01:31.795611 7333 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:31.796464 7333 heartbeater.cc:511] Master 127.30.247.62:43329 requested a full tablet report, sending...
I20260925 05:01:31.796656 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 7198
I20260925 05:01:31.796813 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance
I20260925 05:01:31.797421 31708 external_mini_cluster.cc:1428] Setting key 9ce8bcfbccc0bf2e6ddb6b137bdf7cbc
I20260925 05:01:31.798272 6848 ts_manager.cc:194] Registered new tserver with Master: d8aa4ea6384a4298a04cb4b471fcad4a (127.30.247.3:34259)
I20260925 05:01:31.799346 6848 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.247.3@KRBTEST.COM'} at 127.30.247.3:60915
I20260925 05:01:31.800379 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 05:01:31 dist-test-slave-5n86 krb5kdc[6803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312491, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 05:01:31 dist-test-slave-5n86 krb5kdc[6803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:31 dist-test-slave-5n86 krb5kdc[6803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:32.279445 7194 heartbeater.cc:503] Master 127.30.247.62:43329 was elected leader, sending a full tablet report...
I20260925 05:01:32.801834 7333 heartbeater.cc:503] Master 127.30.247.62:43329 was elected leader, sending a full tablet report...
W20260925 05:01:32.891513 7341 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.050s user 0.001s sys 0.015s
W20260925 05:01:32.892794 6943 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:31.836343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51986 (local address 127.30.247.62:43329)
0925 05:01:31.836615 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:31.836621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:31.836775 (+ 154us) server_negotiation.cc:407] Connection header received
0925 05:01:31.836979 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:31.836983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:31.837048 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:31.837134 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:31.838443 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:31.839265 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:31.840467 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:31.840927 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:31.841025 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:32.892492 (+1051467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51986: BlockingRecv error: recv got EOF from 127.0.0.1:51986 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 25 05:01:32 dist-test-slave-5n86 krb5kdc[6803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312491, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM
I20260925 05:01:32.913192 6943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:32.896891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52000 (local address 127.30.247.62:43329)
0925 05:01:32.897088 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:32.897095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:32.897374 (+ 279us) server_negotiation.cc:407] Connection header received
0925 05:01:32.897579 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:32.897585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:32.897679 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:32.897787 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:32.898982 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:32.899705 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:32.901015 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:32.901399 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:32.901522 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:32.908351 (+ 6829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:32.908356 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:32.908360 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:32.908401 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:32.911048 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:32.911644 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:32.911648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:32.911650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:32.911759 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:32.912046 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:32.912050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:32.912052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:32.912285 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:32.912494 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:32.912722 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 05:01:32.912873 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 05:01:32.917168 6847 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52000:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:01:32.923776 6847 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:01:32.945384 7355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:32.935911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34078 (local address 127.30.247.1:36451)
0925 05:01:32.936657 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:32.936668 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:32.937692 (+ 1024us) server_negotiation.cc:407] Connection header received
0925 05:01:32.937794 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:32.937802 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:32.938079 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:32.938889 (+ 810us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:01:32.939103 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:32.940650 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:32.943034 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:32.944201 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:32.944347 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:32.944578 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:32.944936 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 05:01:32.945110 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":435,"threads_started":1}
I20260925 05:01:32.945429 7352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:32.935369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:36451 (local address 127.0.0.1:34078)
0925 05:01:32.937336 (+ 1967us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:32.937353 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:32.937519 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:32.938321 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:32.938328 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:32.938370 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:01:32.938854 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:32.938868 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:32.940779 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:32.940787 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:32.942877 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:32.942894 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:32.944418 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:32.944428 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:32.944674 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:32.944739 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 05:01:32.944864 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 05:01:32.945034 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1836,"mutex_wait_us":23,"spinlock_wait_cycles":4352,"thread_start_us":2300,"threads_started":1}
I20260925 05:01:32.946836 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:32.936998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:34259 (local address 127.0.0.1:58220)
0925 05:01:32.938245 (+ 1247us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:32.938267 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:32.938354 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:32.939626 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:32.939630 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:32.939656 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:01:32.940079 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:32.940092 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:32.941622 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:32.941627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:32.943971 (+ 2344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:32.943986 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:32.946067 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:32.946073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:32.946233 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:32.946269 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 05:01:32.946361 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 05:01:32.946490 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"mutex_wait_us":445,"thread_start_us":524,"threads_started":1}
I20260925 05:01:32.946843 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:32.937310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58220 (local address 127.30.247.3:34259)
0925 05:01:32.938854 (+ 1544us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:32.938862 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:32.938888 (+ 26us) server_negotiation.cc:407] Connection header received
0925 05:01:32.939051 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:32.939057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:32.939268 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:32.939548 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:01:32.940261 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:32.941458 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:32.944153 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:32.945601 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:32.945721 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:32.945913 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:32.946415 (+ 502us) server_negotiation.cc:299] Negotiation successful
0925 05:01:32.946582 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1382,"thread_start_us":1257,"threads_started":1}
I20260925 05:01:32.947834 7353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:32.935081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:41993 (local address 127.0.0.1:45224)
0925 05:01:32.937281 (+ 2200us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:32.937324 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:32.937533 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:32.939224 (+ 1691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:32.939229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:32.939251 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:01:32.940373 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:32.940395 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:32.942376 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:32.942381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:32.944460 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:32.944472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:32.947280 (+ 2808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:32.947288 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:32.947422 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:32.947444 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 05:01:32.947508 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 05:01:32.947588 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1921,"mutex_wait_us":4,"thread_start_us":1980,"threads_started":1}
I20260925 05:01:32.948029 7354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:32.935403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45224 (local address 127.30.247.2:41993)
0925 05:01:32.936195 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:32.936206 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:32.937425 (+ 1219us) server_negotiation.cc:407] Connection header received
0925 05:01:32.938796 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:32.938804 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:32.939034 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:32.939230 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:01:32.940688 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:32.942236 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:32.944653 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:32.946758 (+ 2105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:32.946919 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:32.947131 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:32.947602 (+ 471us) server_negotiation.cc:299] Negotiation successful
0925 05:01:32.947769 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":458,"threads_started":1}
I20260925 05:01:32.949643 6985 tablet_service.cc:1467] Processing CreateTablet for tablet 59def15eb15448bfa3531704c70a3910 (DEFAULT_TABLE table=test-table [id=97012d70c9144c808715d268ab0fc8eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:01:32.951104 6985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59def15eb15448bfa3531704c70a3910. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:32.950827 7263 tablet_service.cc:1467] Processing CreateTablet for tablet 59def15eb15448bfa3531704c70a3910 (DEFAULT_TABLE table=test-table [id=97012d70c9144c808715d268ab0fc8eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:01:32.951279 7124 tablet_service.cc:1467] Processing CreateTablet for tablet 59def15eb15448bfa3531704c70a3910 (DEFAULT_TABLE table=test-table [id=97012d70c9144c808715d268ab0fc8eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:01:32.952126 7263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59def15eb15448bfa3531704c70a3910. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:32.952482 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59def15eb15448bfa3531704c70a3910. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:32.964311 7358 tablet_bootstrap.cc:492] T 59def15eb15448bfa3531704c70a3910 P d8aa4ea6384a4298a04cb4b471fcad4a: Bootstrap starting.
I20260925 05:01:32.966877 7358 tablet_bootstrap.cc:654] T 59def15eb15448bfa3531704c70a3910 P d8aa4ea6384a4298a04cb4b471fcad4a: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:32.967628 7359 tablet_bootstrap.cc:492] T 59def15eb15448bfa3531704c70a3910 P a78bbd0e639a4d2cb96c920d629cf399: Bootstrap starting.
I20260925 05:01:32.968333 7360 tablet_bootstrap.cc:492] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214: Bootstrap starting.
I20260925 05:01:32.968513 7358 log.cc:824] T 59def15eb15448bfa3531704c70a3910 P d8aa4ea6384a4298a04cb4b471fcad4a: Log is configured to *not* fsync() on all Append() calls
I20260925 05:01:32.971015 7358 tablet_bootstrap.cc:492] T 59def15eb15448bfa3531704c70a3910 P d8aa4ea6384a4298a04cb4b471fcad4a: No bootstrap required, opened a new log
I20260925 05:01:32.971273 7358 ts_tablet_manager.cc:1397] T 59def15eb15448bfa3531704c70a3910 P d8aa4ea6384a4298a04cb4b471fcad4a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260925 05:01:32.971692 7359 tablet_bootstrap.cc:654] T 59def15eb15448bfa3531704c70a3910 P a78bbd0e639a4d2cb96c920d629cf399: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:32.972388 7360 tablet_bootstrap.cc:654] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:32.973047 7359 log.cc:824] T 59def15eb15448bfa3531704c70a3910 P a78bbd0e639a4d2cb96c920d629cf399: Log is configured to *not* fsync() on all Append() calls
I20260925 05:01:32.973639 7360 log.cc:824] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214: Log is configured to *not* fsync() on all Append() calls
I20260925 05:01:32.976132 7360 tablet_bootstrap.cc:492] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214: No bootstrap required, opened a new log
I20260925 05:01:32.976380 7360 ts_tablet_manager.cc:1397] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 05:01:32.976598 7359 tablet_bootstrap.cc:492] T 59def15eb15448bfa3531704c70a3910 P a78bbd0e639a4d2cb96c920d629cf399: No bootstrap required, opened a new log
I20260925 05:01:32.976835 7359 ts_tablet_manager.cc:1397] T 59def15eb15448bfa3531704c70a3910 P a78bbd0e639a4d2cb96c920d629cf399: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 05:01:32.981329 7358 raft_consensus.cc:348] T 59def15eb15448bfa3531704c70a3910 P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:32.981750 7358 raft_consensus.cc:374] T 59def15eb15448bfa3531704c70a3910 P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:32.981882 7358 raft_consensus.cc:729] T 59def15eb15448bfa3531704c70a3910 P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8aa4ea6384a4298a04cb4b471fcad4a, State: Initialized, Role: FOLLOWER
I20260925 05:01:32.982478 7358 consensus_queue.cc:260] T 59def15eb15448bfa3531704c70a3910 P d8aa4ea6384a4298a04cb4b471fcad4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:32.984239 7358 ts_tablet_manager.cc:1428] T 59def15eb15448bfa3531704c70a3910 P d8aa4ea6384a4298a04cb4b471fcad4a: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260925 05:01:32.984627 7360 raft_consensus.cc:348] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:32.985054 7360 raft_consensus.cc:374] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:32.985162 7360 raft_consensus.cc:729] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 892c09a35d4248359563299977a06214, State: Initialized, Role: FOLLOWER
I20260925 05:01:32.985708 7360 consensus_queue.cc:260] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:32.987100 7359 raft_consensus.cc:348] T 59def15eb15448bfa3531704c70a3910 P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:32.987519 7359 raft_consensus.cc:374] T 59def15eb15448bfa3531704c70a3910 P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:32.987638 7359 raft_consensus.cc:729] T 59def15eb15448bfa3531704c70a3910 P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a78bbd0e639a4d2cb96c920d629cf399, State: Initialized, Role: FOLLOWER
I20260925 05:01:32.988169 7360 ts_tablet_manager.cc:1428] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260925 05:01:32.988314 7359 consensus_queue.cc:260] T 59def15eb15448bfa3531704c70a3910 P a78bbd0e639a4d2cb96c920d629cf399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:32.989969 7359 ts_tablet_manager.cc:1428] T 59def15eb15448bfa3531704c70a3910 P a78bbd0e639a4d2cb96c920d629cf399: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s
W20260925 05:01:33.020825 7056 tablet.cc:2369] T 59def15eb15448bfa3531704c70a3910 P a78bbd0e639a4d2cb96c920d629cf399: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:01:33.024835 7195 tablet.cc:2369] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:01:33.045524 7334 tablet.cc:2369] T 59def15eb15448bfa3531704c70a3910 P d8aa4ea6384a4298a04cb4b471fcad4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:01:33.156669 7365 raft_consensus.cc:482] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:01:33.156991 7365 raft_consensus.cc:504] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:33.158756 7365 leader_election.cc:288] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a78bbd0e639a4d2cb96c920d629cf399 (127.30.247.1:36451), d8aa4ea6384a4298a04cb4b471fcad4a (127.30.247.3:34259)
I20260925 05:01:33.165580 7355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:33.159279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:58253 (local address 127.30.247.1:36451)
0925 05:01:33.159471 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:33.159477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:33.160408 (+ 931us) server_negotiation.cc:407] Connection header received
0925 05:01:33.160650 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:33.160656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:33.160733 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:33.160830 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:01:33.161396 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.162105 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.163890 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.164687 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.164850 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:33.164929 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:33.165216 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 05:01:33.165332 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 05:01:33.165639 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:33.159271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:59947 (local address 127.30.247.3:34259)
0925 05:01:33.159470 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:33.159477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:33.160481 (+ 1004us) server_negotiation.cc:407] Connection header received
0925 05:01:33.160707 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:33.160712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:33.160789 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:33.160887 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:01:33.161528 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.162217 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.163960 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.164756 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.164930 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:33.165009 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:33.165276 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 05:01:33.165396 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 05:01:33.165733 7368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:33.159295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:36451 (local address 127.30.247.2:58253)
0925 05:01:33.160299 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:33.160315 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:33.160487 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:33.160848 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:33.160853 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:33.160877 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:01:33.161253 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:33.161265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.162236 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.162242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:33.163743 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:33.163755 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.164897 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.164903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:33.165054 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:33.165077 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 05:01:33.165181 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 05:01:33.165301 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"mutex_wait_us":21,"spinlock_wait_cycles":11648,"thread_start_us":604,"threads_started":1}
I20260925 05:01:33.165781 7367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:33.159295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:34259 (local address 127.30.247.2:59947)
0925 05:01:33.160357 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:33.160372 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:33.160520 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:33.160904 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:33.160908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:33.160945 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:01:33.161375 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:33.161384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.162364 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.162369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:33.163810 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:33.163822 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.164983 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.164989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:33.165122 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:33.165144 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 05:01:33.165243 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 05:01:33.165355 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"mutex_wait_us":59,"spinlock_wait_cycles":22400,"thread_start_us":502,"threads_started":1}
I20260925 05:01:33.166843 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59def15eb15448bfa3531704c70a3910" candidate_uuid: "892c09a35d4248359563299977a06214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a78bbd0e639a4d2cb96c920d629cf399" is_pre_election: true
I20260925 05:01:33.166864 7288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59def15eb15448bfa3531704c70a3910" candidate_uuid: "892c09a35d4248359563299977a06214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" is_pre_election: true
I20260925 05:01:33.167374 7288 raft_consensus.cc:2454] T 59def15eb15448bfa3531704c70a3910 P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 892c09a35d4248359563299977a06214 in term 0.
I20260925 05:01:33.167383 7010 raft_consensus.cc:2454] T 59def15eb15448bfa3531704c70a3910 P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 892c09a35d4248359563299977a06214 in term 0.
I20260925 05:01:33.167997 7078 leader_election.cc:302] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 892c09a35d4248359563299977a06214, a78bbd0e639a4d2cb96c920d629cf399, d8aa4ea6384a4298a04cb4b471fcad4a; no voters:
I20260925 05:01:33.168349 7365 raft_consensus.cc:2781] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:01:33.168519 7365 raft_consensus.cc:482] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:01:33.168678 7365 raft_consensus.cc:3037] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:33.170789 7365 raft_consensus.cc:504] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:33.171350 7365 leader_election.cc:288] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214 [CANDIDATE]: Term 1 election: Requested vote from peers a78bbd0e639a4d2cb96c920d629cf399 (127.30.247.1:36451), d8aa4ea6384a4298a04cb4b471fcad4a (127.30.247.3:34259)
I20260925 05:01:33.171777 7288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59def15eb15448bfa3531704c70a3910" candidate_uuid: "892c09a35d4248359563299977a06214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a"
I20260925 05:01:33.171815 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59def15eb15448bfa3531704c70a3910" candidate_uuid: "892c09a35d4248359563299977a06214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a78bbd0e639a4d2cb96c920d629cf399"
I20260925 05:01:33.171967 7288 raft_consensus.cc:3037] T 59def15eb15448bfa3531704c70a3910 P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:33.171988 7010 raft_consensus.cc:3037] T 59def15eb15448bfa3531704c70a3910 P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:33.174288 7010 raft_consensus.cc:2454] T 59def15eb15448bfa3531704c70a3910 P a78bbd0e639a4d2cb96c920d629cf399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 892c09a35d4248359563299977a06214 in term 1.
I20260925 05:01:33.174386 7288 raft_consensus.cc:2454] T 59def15eb15448bfa3531704c70a3910 P d8aa4ea6384a4298a04cb4b471fcad4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 892c09a35d4248359563299977a06214 in term 1.
I20260925 05:01:33.174712 7078 leader_election.cc:302] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 892c09a35d4248359563299977a06214, a78bbd0e639a4d2cb96c920d629cf399; no voters:
I20260925 05:01:33.174978 7365 raft_consensus.cc:2781] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:01:33.175384 7365 raft_consensus.cc:686] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214 [term 1 LEADER]: Becoming Leader. State: Replica: 892c09a35d4248359563299977a06214, State: Running, Role: LEADER
I20260925 05:01:33.175881 7365 consensus_queue.cc:237] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:33.179936 6847 catalog_manager.cc:5495] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214 reported cstate change: term changed from 0 to 1, leader changed from <none> to 892c09a35d4248359563299977a06214 (127.30.247.2). New cstate: current_term: 1 leader_uuid: "892c09a35d4248359563299977a06214" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } health_report { overall_health: UNKNOWN } } }
I20260925 05:01:33.237131 7354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:33.231832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45232 (local address 127.30.247.2:41993)
0925 05:01:33.232023 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:33.232028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:33.232046 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:01:33.232178 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:33.232182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:33.232278 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:33.232408 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:01:33.232779 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.233511 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.234784 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.235211 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.235312 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:33.235731 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:01:33.236620 (+ 889us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:01:33.236803 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:33.236860 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 05:01:33.236933 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 05:01:33.245007 7288 raft_consensus.cc:1261] T 59def15eb15448bfa3531704c70a3910 P d8aa4ea6384a4298a04cb4b471fcad4a [term 1 FOLLOWER]: Refusing update from remote peer 892c09a35d4248359563299977a06214: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:01:33.245076 7010 raft_consensus.cc:1261] T 59def15eb15448bfa3531704c70a3910 P a78bbd0e639a4d2cb96c920d629cf399 [term 1 FOLLOWER]: Refusing update from remote peer 892c09a35d4248359563299977a06214: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:01:33.245976 7369 consensus_queue.cc:1035] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:01:33.246338 7365 consensus_queue.cc:1035] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:01:33.269143 6846 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:01:33.269410 6846 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:01:33.273552 6846 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32cd628ad4834acdba076e9b198ed8dc: Sending DeleteTablet for 3 replicas of tablet 59def15eb15448bfa3531704c70a3910
I20260925 05:01:33.274675 7124 tablet_service.cc:1514] Processing DeleteTablet for tablet 59def15eb15448bfa3531704c70a3910 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:01:33 UTC) from {username='oryx', principal='oryx/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:45224
I20260925 05:01:33.275185 6985 tablet_service.cc:1514] Processing DeleteTablet for tablet 59def15eb15448bfa3531704c70a3910 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:01:33 UTC) from {username='oryx', principal='oryx/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:34078
I20260925 05:01:33.275612 7263 tablet_service.cc:1514] Processing DeleteTablet for tablet 59def15eb15448bfa3531704c70a3910 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:01:33 UTC) from {username='oryx', principal='oryx/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:58220
I20260925 05:01:33.278102 6846 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52000:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:01:33.279399 6846 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:01:33.288005 7123 tablet_service.cc:1467] Processing CreateTablet for tablet 1dfe9a75f51a4f6990dd7e140b56c5a4 (DEFAULT_TABLE table=test-table [id=32615e77c79647aa98f6dec952101209]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:01:33.288152 6984 tablet_service.cc:1467] Processing CreateTablet for tablet 1dfe9a75f51a4f6990dd7e140b56c5a4 (DEFAULT_TABLE table=test-table [id=32615e77c79647aa98f6dec952101209]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:01:33.288491 7262 tablet_service.cc:1467] Processing CreateTablet for tablet 1dfe9a75f51a4f6990dd7e140b56c5a4 (DEFAULT_TABLE table=test-table [id=32615e77c79647aa98f6dec952101209]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:01:33.288620 7123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dfe9a75f51a4f6990dd7e140b56c5a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:33.288678 6984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dfe9a75f51a4f6990dd7e140b56c5a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:33.289011 7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dfe9a75f51a4f6990dd7e140b56c5a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:33.291191 7381 tablet_replica.cc:330] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214: stopping tablet replica
I20260925 05:01:33.291608 7381 raft_consensus.cc:2229] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:01:33.292240 7381 raft_consensus.cc:2258] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:01:33.294055 7381 ts_tablet_manager.cc:1905] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:01:33.295068 7359 tablet_bootstrap.cc:492] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P a78bbd0e639a4d2cb96c920d629cf399: Bootstrap starting.
I20260925 05:01:33.297127 7359 tablet_bootstrap.cc:654] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P a78bbd0e639a4d2cb96c920d629cf399: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:33.298213 7360 tablet_bootstrap.cc:492] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214: Bootstrap starting.
I20260925 05:01:33.299178 7358 tablet_bootstrap.cc:492] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P d8aa4ea6384a4298a04cb4b471fcad4a: Bootstrap starting.
I20260925 05:01:33.299772 7360 tablet_bootstrap.cc:654] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:33.301234 7358 tablet_bootstrap.cc:654] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P d8aa4ea6384a4298a04cb4b471fcad4a: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:33.305576 7383 tablet_replica.cc:330] T 59def15eb15448bfa3531704c70a3910 P d8aa4ea6384a4298a04cb4b471fcad4a: stopping tablet replica
I20260925 05:01:33.307199 7381 ts_tablet_manager.cc:1918] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:01:33.309247 7383 raft_consensus.cc:2229] T 59def15eb15448bfa3531704c70a3910 P d8aa4ea6384a4298a04cb4b471fcad4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:01:33.309931 7383 raft_consensus.cc:2258] T 59def15eb15448bfa3531704c70a3910 P d8aa4ea6384a4298a04cb4b471fcad4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:01:33.310583 7381 log.cc:1196] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-1/wal/wals/59def15eb15448bfa3531704c70a3910
I20260925 05:01:33.311266 7381 ts_tablet_manager.cc:1939] T 59def15eb15448bfa3531704c70a3910 P 892c09a35d4248359563299977a06214: Deleting consensus metadata
I20260925 05:01:33.313184 7359 tablet_bootstrap.cc:492] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P a78bbd0e639a4d2cb96c920d629cf399: No bootstrap required, opened a new log
I20260925 05:01:33.313388 7359 ts_tablet_manager.cc:1397] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P a78bbd0e639a4d2cb96c920d629cf399: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.004s
I20260925 05:01:33.313443 6834 catalog_manager.cc:4841] TS 892c09a35d4248359563299977a06214 (127.30.247.2:41993): tablet 59def15eb15448bfa3531704c70a3910 (table test-table [id=97012d70c9144c808715d268ab0fc8eb]) successfully deleted
I20260925 05:01:33.314365 7359 raft_consensus.cc:348] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:33.314661 7359 raft_consensus.cc:374] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:33.314771 7359 raft_consensus.cc:729] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a78bbd0e639a4d2cb96c920d629cf399, State: Initialized, Role: FOLLOWER
I20260925 05:01:33.315064 7359 consensus_queue.cc:260] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P a78bbd0e639a4d2cb96c920d629cf399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:33.315774 7359 ts_tablet_manager.cc:1428] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P a78bbd0e639a4d2cb96c920d629cf399: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:01:33.316444 7383 ts_tablet_manager.cc:1905] T 59def15eb15448bfa3531704c70a3910 P d8aa4ea6384a4298a04cb4b471fcad4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:01:33.317611 7382 tablet_replica.cc:330] T 59def15eb15448bfa3531704c70a3910 P a78bbd0e639a4d2cb96c920d629cf399: stopping tablet replica
I20260925 05:01:33.319502 7358 tablet_bootstrap.cc:492] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P d8aa4ea6384a4298a04cb4b471fcad4a: No bootstrap required, opened a new log
I20260925 05:01:33.319646 7358 ts_tablet_manager.cc:1397] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P d8aa4ea6384a4298a04cb4b471fcad4a: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.007s
I20260925 05:01:33.320456 7358 raft_consensus.cc:348] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:33.320706 7358 raft_consensus.cc:374] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:33.320806 7358 raft_consensus.cc:729] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8aa4ea6384a4298a04cb4b471fcad4a, State: Initialized, Role: FOLLOWER
I20260925 05:01:33.321056 7358 consensus_queue.cc:260] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P d8aa4ea6384a4298a04cb4b471fcad4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:33.321605 7358 ts_tablet_manager.cc:1428] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P d8aa4ea6384a4298a04cb4b471fcad4a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:01:33.322917 7382 raft_consensus.cc:2229] T 59def15eb15448bfa3531704c70a3910 P a78bbd0e639a4d2cb96c920d629cf399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:01:33.323392 7383 ts_tablet_manager.cc:1918] T 59def15eb15448bfa3531704c70a3910 P d8aa4ea6384a4298a04cb4b471fcad4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:01:33.323410 7382 raft_consensus.cc:2258] T 59def15eb15448bfa3531704c70a3910 P a78bbd0e639a4d2cb96c920d629cf399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:01:33.323614 7383 log.cc:1196] T 59def15eb15448bfa3531704c70a3910 P d8aa4ea6384a4298a04cb4b471fcad4a: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-2/wal/wals/59def15eb15448bfa3531704c70a3910
I20260925 05:01:33.324182 7383 ts_tablet_manager.cc:1939] T 59def15eb15448bfa3531704c70a3910 P d8aa4ea6384a4298a04cb4b471fcad4a: Deleting consensus metadata
I20260925 05:01:33.325836 6832 catalog_manager.cc:4841] TS d8aa4ea6384a4298a04cb4b471fcad4a (127.30.247.3:34259): tablet 59def15eb15448bfa3531704c70a3910 (table test-table [id=97012d70c9144c808715d268ab0fc8eb]) successfully deleted
I20260925 05:01:33.326597 7360 tablet_bootstrap.cc:492] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214: No bootstrap required, opened a new log
I20260925 05:01:33.326785 7360 ts_tablet_manager.cc:1397] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.009s
I20260925 05:01:33.327356 7382 ts_tablet_manager.cc:1905] T 59def15eb15448bfa3531704c70a3910 P a78bbd0e639a4d2cb96c920d629cf399: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:01:33.327618 7360 raft_consensus.cc:348] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:33.327881 7360 raft_consensus.cc:374] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:33.328019 7360 raft_consensus.cc:729] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 892c09a35d4248359563299977a06214, State: Initialized, Role: FOLLOWER
I20260925 05:01:33.328315 7360 consensus_queue.cc:260] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:33.328939 7360 ts_tablet_manager.cc:1428] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 05:01:33.334075 7382 ts_tablet_manager.cc:1918] T 59def15eb15448bfa3531704c70a3910 P a78bbd0e639a4d2cb96c920d629cf399: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:01:33.334216 7382 log.cc:1196] T 59def15eb15448bfa3531704c70a3910 P a78bbd0e639a4d2cb96c920d629cf399: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-0/wal/wals/59def15eb15448bfa3531704c70a3910
I20260925 05:01:33.334734 7382 ts_tablet_manager.cc:1939] T 59def15eb15448bfa3531704c70a3910 P a78bbd0e639a4d2cb96c920d629cf399: Deleting consensus metadata
I20260925 05:01:33.336378 6835 catalog_manager.cc:4841] TS a78bbd0e639a4d2cb96c920d629cf399 (127.30.247.1:36451): tablet 59def15eb15448bfa3531704c70a3910 (table test-table [id=97012d70c9144c808715d268ab0fc8eb]) successfully deleted
I20260925 05:01:33.436786 7369 raft_consensus.cc:482] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:01:33.437019 7369 raft_consensus.cc:504] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:33.437772 7369 leader_election.cc:288] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a78bbd0e639a4d2cb96c920d629cf399 (127.30.247.1:36451), d8aa4ea6384a4298a04cb4b471fcad4a (127.30.247.3:34259)
I20260925 05:01:33.438323 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfe9a75f51a4f6990dd7e140b56c5a4" candidate_uuid: "892c09a35d4248359563299977a06214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a78bbd0e639a4d2cb96c920d629cf399" is_pre_election: true
I20260925 05:01:33.438367 7288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfe9a75f51a4f6990dd7e140b56c5a4" candidate_uuid: "892c09a35d4248359563299977a06214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" is_pre_election: true
I20260925 05:01:33.438565 7010 raft_consensus.cc:2454] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 892c09a35d4248359563299977a06214 in term 0.
I20260925 05:01:33.438642 7288 raft_consensus.cc:2454] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 892c09a35d4248359563299977a06214 in term 0.
I20260925 05:01:33.439072 7078 leader_election.cc:302] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 892c09a35d4248359563299977a06214, a78bbd0e639a4d2cb96c920d629cf399; no voters:
I20260925 05:01:33.439356 7369 raft_consensus.cc:2781] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:01:33.439453 7369 raft_consensus.cc:482] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:01:33.439584 7369 raft_consensus.cc:3037] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:33.441782 7369 raft_consensus.cc:504] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:33.442366 7369 leader_election.cc:288] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214 [CANDIDATE]: Term 1 election: Requested vote from peers a78bbd0e639a4d2cb96c920d629cf399 (127.30.247.1:36451), d8aa4ea6384a4298a04cb4b471fcad4a (127.30.247.3:34259)
I20260925 05:01:33.442742 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfe9a75f51a4f6990dd7e140b56c5a4" candidate_uuid: "892c09a35d4248359563299977a06214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a78bbd0e639a4d2cb96c920d629cf399"
I20260925 05:01:33.442826 7288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfe9a75f51a4f6990dd7e140b56c5a4" candidate_uuid: "892c09a35d4248359563299977a06214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a"
I20260925 05:01:33.442919 7010 raft_consensus.cc:3037] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:33.443013 7288 raft_consensus.cc:3037] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:33.445057 7010 raft_consensus.cc:2454] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P a78bbd0e639a4d2cb96c920d629cf399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 892c09a35d4248359563299977a06214 in term 1.
I20260925 05:01:33.445451 7078 leader_election.cc:302] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 892c09a35d4248359563299977a06214, a78bbd0e639a4d2cb96c920d629cf399; no voters:
I20260925 05:01:33.445714 7369 raft_consensus.cc:2781] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:01:33.445894 7369 raft_consensus.cc:686] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214 [term 1 LEADER]: Becoming Leader. State: Replica: 892c09a35d4248359563299977a06214, State: Running, Role: LEADER
I20260925 05:01:33.445933 7288 raft_consensus.cc:2454] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P d8aa4ea6384a4298a04cb4b471fcad4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 892c09a35d4248359563299977a06214 in term 1.
I20260925 05:01:33.446151 7369 consensus_queue.cc:237] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:33.449200 6847 catalog_manager.cc:5495] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214 reported cstate change: term changed from 0 to 1, leader changed from <none> to 892c09a35d4248359563299977a06214 (127.30.247.2). New cstate: current_term: 1 leader_uuid: "892c09a35d4248359563299977a06214" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } health_report { overall_health: UNKNOWN } } }
Sep 25 05:01:33 dist-test-slave-5n86 krb5kdc[6803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312490, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.62@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM
I20260925 05:01:33.492086 7394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:33.478792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:43329 (local address 127.0.0.1:52006)
0925 05:01:33.479609 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:33.479628 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:33.479786 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:33.480097 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:33.480101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:33.480834 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:33.481013 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:33.481021 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.482005 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.482013 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:33.482934 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:33.482943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.483647 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.483653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:33.483815 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:33.484479 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:33.484496 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:33.486778 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:33.489669 (+ 2891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:33.489680 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:33.489687 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:33.490055 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:33.490388 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:33.490395 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:33.490399 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:33.490530 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:33.491016 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:33.491024 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:33.491184 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 05:01:33.491347 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 05:01:33.491628 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":549,"threads_started":1}
I20260925 05:01:33.492107 7393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:33.478788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52006 (local address 127.30.247.62:43329)
0925 05:01:33.479482 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:33.479488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:33.479710 (+ 222us) server_negotiation.cc:407] Connection header received
0925 05:01:33.479913 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:33.479917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:33.479974 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:33.480078 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:33.481153 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.481884 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.483073 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.483489 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.483616 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:33.486954 (+ 3338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:33.486959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:33.486962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:33.486988 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:33.489504 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:33.490167 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:33.490175 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:33.490179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:33.490286 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:33.490635 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:33.490640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:33.490644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:33.490889 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:33.491036 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:33.491434 (+ 398us) server_negotiation.cc:299] Negotiation successful
0925 05:01:33.491746 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":486,"thread_start_us":328,"threads_started":1}
I20260925 05:01:33.496876 6847 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:52006:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 05:01:33.497952 6847 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:01:33.504779 6985 tablet_service.cc:1467] Processing CreateTablet for tablet 592c384f708946338a0b662645ed61bb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3a83b7c143d14839bb02ea81e34bfb0c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:01:33.505155 7263 tablet_service.cc:1467] Processing CreateTablet for tablet 592c384f708946338a0b662645ed61bb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3a83b7c143d14839bb02ea81e34bfb0c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:01:33.505393 6985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 592c384f708946338a0b662645ed61bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:33.505681 7263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 592c384f708946338a0b662645ed61bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:33.509078 7123 tablet_service.cc:1467] Processing CreateTablet for tablet 592c384f708946338a0b662645ed61bb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3a83b7c143d14839bb02ea81e34bfb0c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:01:33.509629 7123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 592c384f708946338a0b662645ed61bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:33.512486 7359 tablet_bootstrap.cc:492] T 592c384f708946338a0b662645ed61bb P a78bbd0e639a4d2cb96c920d629cf399: Bootstrap starting.
I20260925 05:01:33.512519 7358 tablet_bootstrap.cc:492] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a: Bootstrap starting.
I20260925 05:01:33.514166 7358 tablet_bootstrap.cc:654] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:33.514207 7359 tablet_bootstrap.cc:654] T 592c384f708946338a0b662645ed61bb P a78bbd0e639a4d2cb96c920d629cf399: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:33.515970 7359 tablet_bootstrap.cc:492] T 592c384f708946338a0b662645ed61bb P a78bbd0e639a4d2cb96c920d629cf399: No bootstrap required, opened a new log
I20260925 05:01:33.516152 7359 ts_tablet_manager.cc:1397] T 592c384f708946338a0b662645ed61bb P a78bbd0e639a4d2cb96c920d629cf399: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 05:01:33.516840 7359 raft_consensus.cc:348] T 592c384f708946338a0b662645ed61bb P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:33.517083 7359 raft_consensus.cc:374] T 592c384f708946338a0b662645ed61bb P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:33.517182 7359 raft_consensus.cc:729] T 592c384f708946338a0b662645ed61bb P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a78bbd0e639a4d2cb96c920d629cf399, State: Initialized, Role: FOLLOWER
I20260925 05:01:33.517422 7359 consensus_queue.cc:260] T 592c384f708946338a0b662645ed61bb P a78bbd0e639a4d2cb96c920d629cf399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:33.518055 7359 ts_tablet_manager.cc:1428] T 592c384f708946338a0b662645ed61bb P a78bbd0e639a4d2cb96c920d629cf399: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:01:33.518105 7358 tablet_bootstrap.cc:492] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a: No bootstrap required, opened a new log
I20260925 05:01:33.517910 7366 tablet_replica.cc:439] T 592c384f708946338a0b662645ed61bb P a78bbd0e639a4d2cb96c920d629cf399: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } } }
I20260925 05:01:33.518246 7358 ts_tablet_manager.cc:1397] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260925 05:01:33.518289 7366 tablet_replica.cc:442] T 592c384f708946338a0b662645ed61bb P a78bbd0e639a4d2cb96c920d629cf399: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:01:33.518627 7360 tablet_bootstrap.cc:492] T 592c384f708946338a0b662645ed61bb P 892c09a35d4248359563299977a06214: Bootstrap starting.
I20260925 05:01:33.518911 7358 raft_consensus.cc:348] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:33.519140 7358 raft_consensus.cc:374] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:33.519217 7358 raft_consensus.cc:729] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8aa4ea6384a4298a04cb4b471fcad4a, State: Initialized, Role: FOLLOWER
I20260925 05:01:33.519462 7358 consensus_queue.cc:260] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:33.519870 7364 tablet_replica.cc:439] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } } }
I20260925 05:01:33.520220 7364 tablet_replica.cc:442] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:01:33.520295 7360 tablet_bootstrap.cc:654] T 592c384f708946338a0b662645ed61bb P 892c09a35d4248359563299977a06214: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:33.521183 7358 ts_tablet_manager.cc:1428] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 05:01:33.523715 7360 tablet_bootstrap.cc:492] T 592c384f708946338a0b662645ed61bb P 892c09a35d4248359563299977a06214: No bootstrap required, opened a new log
I20260925 05:01:33.523895 7360 ts_tablet_manager.cc:1397] T 592c384f708946338a0b662645ed61bb P 892c09a35d4248359563299977a06214: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260925 05:01:33.524526 7360 raft_consensus.cc:348] T 592c384f708946338a0b662645ed61bb P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:33.524782 7360 raft_consensus.cc:374] T 592c384f708946338a0b662645ed61bb P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:33.524880 7360 raft_consensus.cc:729] T 592c384f708946338a0b662645ed61bb P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 892c09a35d4248359563299977a06214, State: Initialized, Role: FOLLOWER
I20260925 05:01:33.525130 7360 consensus_queue.cc:260] T 592c384f708946338a0b662645ed61bb P 892c09a35d4248359563299977a06214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:33.525727 7360 ts_tablet_manager.cc:1428] T 592c384f708946338a0b662645ed61bb P 892c09a35d4248359563299977a06214: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260925 05:01:33.525571 7369 tablet_replica.cc:439] T 592c384f708946338a0b662645ed61bb P 892c09a35d4248359563299977a06214: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } } }
I20260925 05:01:33.525916 7369 tablet_replica.cc:442] T 592c384f708946338a0b662645ed61bb P 892c09a35d4248359563299977a06214: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:01:33.623040 7364 raft_consensus.cc:482] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:01:33.623296 7364 raft_consensus.cc:504] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:33.625296 7364 leader_election.cc:288] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a78bbd0e639a4d2cb96c920d629cf399 (127.30.247.1:36451), 892c09a35d4248359563299977a06214 (127.30.247.2:41993)
I20260925 05:01:33.633287 7354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:33.626018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:34381 (local address 127.30.247.2:41993)
0925 05:01:33.626418 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:33.626427 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:33.626455 (+ 28us) server_negotiation.cc:407] Connection header received
0925 05:01:33.626639 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:33.626645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:33.626720 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:33.626781 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:01:33.627722 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.628680 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.631137 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.632122 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.632226 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:33.632280 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:33.632799 (+ 519us) server_negotiation.cc:299] Negotiation successful
0925 05:01:33.632947 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20260925 05:01:33.633834 7355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:33.625694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:53003 (local address 127.30.247.1:36451)
0925 05:01:33.625868 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:33.625874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:33.627061 (+ 1187us) server_negotiation.cc:407] Connection header received
0925 05:01:33.627269 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:33.627281 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:33.627337 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:33.627443 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:01:33.628102 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.629232 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.631525 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.632825 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.632995 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:33.633084 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:33.633417 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 05:01:33.633548 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 05:01:33.634024 7399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:33.625654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:36451 (local address 127.30.247.3:53003)
0925 05:01:33.626937 (+ 1283us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:33.626955 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:33.627117 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:33.627466 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:33.627470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:33.627493 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:01:33.627895 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:33.627907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.629374 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.629379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:33.631375 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:33.631389 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.633052 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.633059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:33.633229 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:33.633274 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 05:01:33.633373 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 05:01:33.633594 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1174,"mutex_wait_us":27,"spinlock_wait_cycles":2816,"thread_start_us":1012,"threads_started":1}
I20260925 05:01:33.634475 7398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:33.625623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:41993 (local address 127.30.247.3:34381)
0925 05:01:33.626299 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:33.626318 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:33.626452 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:33.627005 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:33.627009 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:33.627051 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:01:33.627476 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:33.627490 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.628811 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.628816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:33.630917 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:33.630957 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.632441 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.632445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:33.632609 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:33.632648 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 05:01:33.633672 (+ 1024us) client_negotiation.cc:240] Negotiation successful
0925 05:01:33.633802 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":1,"spinlock_wait_cycles":7808,"thread_start_us":399,"threads_started":1}
I20260925 05:01:33.634642 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "592c384f708946338a0b662645ed61bb" candidate_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a78bbd0e639a4d2cb96c920d629cf399" is_pre_election: true
I20260925 05:01:33.634977 7010 raft_consensus.cc:2454] T 592c384f708946338a0b662645ed61bb P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8aa4ea6384a4298a04cb4b471fcad4a in term 0.
I20260925 05:01:33.635351 7148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "592c384f708946338a0b662645ed61bb" candidate_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "892c09a35d4248359563299977a06214" is_pre_election: true
I20260925 05:01:33.635542 7217 leader_election.cc:302] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a78bbd0e639a4d2cb96c920d629cf399, d8aa4ea6384a4298a04cb4b471fcad4a; no voters:
I20260925 05:01:33.635842 7148 raft_consensus.cc:2454] T 592c384f708946338a0b662645ed61bb P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8aa4ea6384a4298a04cb4b471fcad4a in term 0.
I20260925 05:01:33.635964 7364 raft_consensus.cc:2781] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:01:33.636101 7364 raft_consensus.cc:482] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:01:33.636204 7364 raft_consensus.cc:3037] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:33.639454 7364 raft_consensus.cc:504] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:33.640158 7364 leader_election.cc:288] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a [CANDIDATE]: Term 1 election: Requested vote from peers a78bbd0e639a4d2cb96c920d629cf399 (127.30.247.1:36451), 892c09a35d4248359563299977a06214 (127.30.247.2:41993)
I20260925 05:01:33.640556 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "592c384f708946338a0b662645ed61bb" candidate_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a78bbd0e639a4d2cb96c920d629cf399"
I20260925 05:01:33.640630 7148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "592c384f708946338a0b662645ed61bb" candidate_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "892c09a35d4248359563299977a06214"
I20260925 05:01:33.640779 7010 raft_consensus.cc:3037] T 592c384f708946338a0b662645ed61bb P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:33.640846 7148 raft_consensus.cc:3037] T 592c384f708946338a0b662645ed61bb P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:33.643231 7148 raft_consensus.cc:2454] T 592c384f708946338a0b662645ed61bb P 892c09a35d4248359563299977a06214 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8aa4ea6384a4298a04cb4b471fcad4a in term 1.
I20260925 05:01:33.643697 7216 leader_election.cc:302] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 892c09a35d4248359563299977a06214, d8aa4ea6384a4298a04cb4b471fcad4a; no voters:
I20260925 05:01:33.643723 7010 raft_consensus.cc:2454] T 592c384f708946338a0b662645ed61bb P a78bbd0e639a4d2cb96c920d629cf399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8aa4ea6384a4298a04cb4b471fcad4a in term 1.
I20260925 05:01:33.643993 7364 raft_consensus.cc:2781] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:01:33.644560 7364 raft_consensus.cc:686] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a [term 1 LEADER]: Becoming Leader. State: Replica: d8aa4ea6384a4298a04cb4b471fcad4a, State: Running, Role: LEADER
I20260925 05:01:33.645185 7364 consensus_queue.cc:237] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:33.647152 7400 tablet_replica.cc:439] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a: TxnStatusTablet state changed. Reason: New leader d8aa4ea6384a4298a04cb4b471fcad4a. Latest consensus state: current_term: 1 leader_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } } }
I20260925 05:01:33.647614 7400 tablet_replica.cc:442] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:01:33.648357 7402 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 05:01:33.648617 7402 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 05:01:33.648972 6846 catalog_manager.cc:5495] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a reported cstate change: term changed from 0 to 1, leader changed from <none> to d8aa4ea6384a4298a04cb4b471fcad4a (127.30.247.3). New cstate: current_term: 1 leader_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } health_report { overall_health: HEALTHY } } }
I20260925 05:01:33.685626 7394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:33.677854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:34259 (local address 127.0.0.1:58232)
0925 05:01:33.678203 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:33.678225 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:33.678350 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:33.679287 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:33.679292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:33.679307 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:01:33.679500 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:33.679514 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.681063 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.681071 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:33.682675 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:33.682690 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.683642 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.683648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:33.683863 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:33.683931 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:01:33.685007 (+ 1076us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:01:33.685045 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 05:01:33.685138 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 05:01:33.685297 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189}
I20260925 05:01:33.685648 7405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:33.677931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58232 (local address 127.30.247.3:34259)
0925 05:01:33.678859 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:33.678865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:33.678883 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:01:33.678981 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:33.678986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:33.679080 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:33.679238 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:01:33.679679 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.680904 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.682833 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.683479 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.683607 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:33.684137 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:01:33.684883 (+ 746us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:01:33.685033 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:33.685189 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 05:01:33.685268 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":803,"thread_start_us":655,"threads_started":1}
I20260925 05:01:33.689914 7010 raft_consensus.cc:1261] T 592c384f708946338a0b662645ed61bb P a78bbd0e639a4d2cb96c920d629cf399 [term 1 FOLLOWER]: Refusing update from remote peer d8aa4ea6384a4298a04cb4b471fcad4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:01:33.690172 7148 raft_consensus.cc:1261] T 592c384f708946338a0b662645ed61bb P 892c09a35d4248359563299977a06214 [term 1 FOLLOWER]: Refusing update from remote peer d8aa4ea6384a4298a04cb4b471fcad4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:01:33.690582 7364 consensus_queue.cc:1035] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:01:33.690917 7400 consensus_queue.cc:1035] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:01:33.698983 7369 tablet_replica.cc:439] T 592c384f708946338a0b662645ed61bb P 892c09a35d4248359563299977a06214: TxnStatusTablet state changed. Reason: New leader d8aa4ea6384a4298a04cb4b471fcad4a. Latest consensus state: current_term: 1 leader_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } } }
I20260925 05:01:33.699271 7369 tablet_replica.cc:442] T 592c384f708946338a0b662645ed61bb P 892c09a35d4248359563299977a06214: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:01:33.703222 7364 tablet_replica.cc:439] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } } }
I20260925 05:01:33.703481 7364 tablet_replica.cc:442] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:01:33.705746 7372 mvcc.cc:205] Tried to move back new op lower bound from 7333119974146621440 to 7333119973974224896. Current Snapshot: MvccSnapshot[applied={T|T < 7333119974146621440}]
I20260925 05:01:33.706321 7369 tablet_replica.cc:439] T 592c384f708946338a0b662645ed61bb P 892c09a35d4248359563299977a06214: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } } }
I20260925 05:01:33.706560 7369 tablet_replica.cc:442] T 592c384f708946338a0b662645ed61bb P 892c09a35d4248359563299977a06214: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:01:33.707141 7373 mvcc.cc:205] Tried to move back new op lower bound from 7333119974146621440 to 7333119973974224896. Current Snapshot: MvccSnapshot[applied={T|T < 7333119974146621440 or (T in {7333119974146621440})}]
I20260925 05:01:33.712574 7366 tablet_replica.cc:439] T 592c384f708946338a0b662645ed61bb P a78bbd0e639a4d2cb96c920d629cf399: TxnStatusTablet state changed. Reason: New leader d8aa4ea6384a4298a04cb4b471fcad4a. Latest consensus state: current_term: 1 leader_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } } }
I20260925 05:01:33.712826 7366 tablet_replica.cc:442] T 592c384f708946338a0b662645ed61bb P a78bbd0e639a4d2cb96c920d629cf399: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:01:33.715518 7400 tablet_replica.cc:439] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } } }
I20260925 05:01:33.715764 7400 tablet_replica.cc:442] T 592c384f708946338a0b662645ed61bb P d8aa4ea6384a4298a04cb4b471fcad4a: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:01:33.716976 7374 mvcc.cc:205] Tried to move back new op lower bound from 7333119974146621440 to 7333119973974224896. Current Snapshot: MvccSnapshot[applied={T|T < 7333119974146621440}]
I20260925 05:01:33.718058 7366 tablet_replica.cc:439] T 592c384f708946338a0b662645ed61bb P a78bbd0e639a4d2cb96c920d629cf399: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } } }
I20260925 05:01:33.718247 7366 tablet_replica.cc:442] T 592c384f708946338a0b662645ed61bb P a78bbd0e639a4d2cb96c920d629cf399: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:01:33.734750 7405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:33.727141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:59835 (local address 127.30.247.3:34259)
0925 05:01:33.727318 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:33.727325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:33.728001 (+ 676us) server_negotiation.cc:407] Connection header received
0925 05:01:33.728184 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:33.728190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:33.728266 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:33.728351 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:01:33.729214 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.730341 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.732663 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.733344 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.733469 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:33.734006 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:01:33.734139 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:01:33.734224 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:33.734369 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 05:01:33.734443 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 05:01:33.734906 7410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:33.727198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:34259 (local address 127.30.247.2:59835)
0925 05:01:33.727886 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:33.727905 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:33.728057 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:33.728382 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:33.728388 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:33.728401 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:01:33.729033 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:33.729050 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.730499 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.730505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:33.732433 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:33.732451 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.733536 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.733549 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:33.733743 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:33.733813 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:01:33.734245 (+ 432us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:01:33.734256 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 05:01:33.734337 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 05:01:33.734480 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":318,"threads_started":1}
I20260925 05:01:33.745512 7010 raft_consensus.cc:1261] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P a78bbd0e639a4d2cb96c920d629cf399 [term 1 FOLLOWER]: Refusing update from remote peer 892c09a35d4248359563299977a06214: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:01:33.746105 7365 consensus_queue.cc:1035] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:01:33.747237 7288 raft_consensus.cc:1261] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P d8aa4ea6384a4298a04cb4b471fcad4a [term 1 FOLLOWER]: Refusing update from remote peer 892c09a35d4248359563299977a06214: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:01:33.749312 7365 consensus_queue.cc:1035] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:01:33.781174 7354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:33.774016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:35257 (local address 127.30.247.2:41993)
0925 05:01:33.774173 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:33.774179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:33.774754 (+ 575us) server_negotiation.cc:407] Connection header received
0925 05:01:33.774994 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:33.775000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:33.775082 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:33.775134 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:01:33.775836 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.776953 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.779106 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.779725 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.779806 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:33.780445 (+ 639us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:01:33.780580 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:01:33.780635 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:33.780828 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 05:01:33.780902 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 05:01:33.781880 7416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:33.773962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:41993 (local address 127.30.247.3:35257)
0925 05:01:33.774662 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:33.774687 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:33.774848 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:33.775299 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:33.775304 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:33.775316 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:01:33.775668 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:33.775678 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.777121 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.777126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:33.778939 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:33.778952 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.779991 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.779996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:33.780181 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:33.780245 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:01:33.780712 (+ 467us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:01:33.780721 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 05:01:33.781376 (+ 655us) client_negotiation.cc:240] Negotiation successful
0925 05:01:33.781512 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":357,"threads_started":1}
I20260925 05:01:33.837726 6846 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:01:33.837970 6846 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:01:33.841051 6846 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32cd628ad4834acdba076e9b198ed8dc: Sending DeleteTablet for 3 replicas of tablet 1dfe9a75f51a4f6990dd7e140b56c5a4
I20260925 05:01:33.841900 6985 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dfe9a75f51a4f6990dd7e140b56c5a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:01:33 UTC) from {username='oryx', principal='oryx/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:34078
I20260925 05:01:33.842023 7123 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dfe9a75f51a4f6990dd7e140b56c5a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:01:33 UTC) from {username='oryx', principal='oryx/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:45224
I20260925 05:01:33.842072 7263 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dfe9a75f51a4f6990dd7e140b56c5a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:01:33 UTC) from {username='oryx', principal='oryx/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:58220
I20260925 05:01:33.857513 7421 tablet_replica.cc:330] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P d8aa4ea6384a4298a04cb4b471fcad4a: stopping tablet replica
I20260925 05:01:33.858431 7421 raft_consensus.cc:2229] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P d8aa4ea6384a4298a04cb4b471fcad4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:01:33.859187 7421 raft_consensus.cc:2258] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P d8aa4ea6384a4298a04cb4b471fcad4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:01:33.860471 7419 tablet_replica.cc:330] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P a78bbd0e639a4d2cb96c920d629cf399: stopping tablet replica
I20260925 05:01:33.860765 7419 raft_consensus.cc:2229] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P a78bbd0e639a4d2cb96c920d629cf399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:01:33.861058 7419 raft_consensus.cc:2258] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P a78bbd0e639a4d2cb96c920d629cf399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:01:33.861318 7421 ts_tablet_manager.cc:1905] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P d8aa4ea6384a4298a04cb4b471fcad4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:01:33.862356 7419 ts_tablet_manager.cc:1905] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P a78bbd0e639a4d2cb96c920d629cf399: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:01:33.868975 7420 tablet_replica.cc:330] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214: stopping tablet replica
I20260925 05:01:33.869285 7420 raft_consensus.cc:2229] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:01:33.869760 7420 raft_consensus.cc:2258] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:01:33.869860 7421 ts_tablet_manager.cc:1918] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P d8aa4ea6384a4298a04cb4b471fcad4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:01:33.870069 7421 log.cc:1196] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P d8aa4ea6384a4298a04cb4b471fcad4a: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-2/wal/wals/1dfe9a75f51a4f6990dd7e140b56c5a4
I20260925 05:01:33.870510 7419 ts_tablet_manager.cc:1918] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P a78bbd0e639a4d2cb96c920d629cf399: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:01:33.870656 7421 ts_tablet_manager.cc:1939] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P d8aa4ea6384a4298a04cb4b471fcad4a: Deleting consensus metadata
I20260925 05:01:33.870743 7419 log.cc:1196] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P a78bbd0e639a4d2cb96c920d629cf399: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-0/wal/wals/1dfe9a75f51a4f6990dd7e140b56c5a4
I20260925 05:01:33.870991 7420 ts_tablet_manager.cc:1905] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:01:33.871296 7419 ts_tablet_manager.cc:1939] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P a78bbd0e639a4d2cb96c920d629cf399: Deleting consensus metadata
I20260925 05:01:33.871829 6832 catalog_manager.cc:4841] TS d8aa4ea6384a4298a04cb4b471fcad4a (127.30.247.3:34259): tablet 1dfe9a75f51a4f6990dd7e140b56c5a4 (table test-table [id=32615e77c79647aa98f6dec952101209]) successfully deleted
I20260925 05:01:33.872907 6835 catalog_manager.cc:4841] TS a78bbd0e639a4d2cb96c920d629cf399 (127.30.247.1:36451): tablet 1dfe9a75f51a4f6990dd7e140b56c5a4 (table test-table [id=32615e77c79647aa98f6dec952101209]) successfully deleted
I20260925 05:01:33.876820 7420 ts_tablet_manager.cc:1918] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:01:33.876987 7420 log.cc:1196] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-1/wal/wals/1dfe9a75f51a4f6990dd7e140b56c5a4
I20260925 05:01:33.877406 7420 ts_tablet_manager.cc:1939] T 1dfe9a75f51a4f6990dd7e140b56c5a4 P 892c09a35d4248359563299977a06214: Deleting consensus metadata
I20260925 05:01:33.878572 6834 catalog_manager.cc:4841] TS 892c09a35d4248359563299977a06214 (127.30.247.2:41993): tablet 1dfe9a75f51a4f6990dd7e140b56c5a4 (table test-table [id=32615e77c79647aa98f6dec952101209]) successfully deleted
Sep 25 05:01:33 dist-test-slave-5n86 krb5kdc[6803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 05:01:33 dist-test-slave-5n86 krb5kdc[6803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:33 dist-test-slave-5n86 krb5kdc[6803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 05:01:33.913827 7393 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:33.901862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52020 (local address 127.30.247.62:43329)
0925 05:01:33.902130 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:33.902137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:33.902487 (+ 350us) server_negotiation.cc:407] Connection header received
0925 05:01:33.902705 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:33.902710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:33.902776 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:33.902868 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:33.904526 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.905315 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.907122 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.907545 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.907657 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:33.913581 (+ 5924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52020: BlockingRecv error: recv got EOF from 127.0.0.1:52020 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 05:01:33 dist-test-slave-5n86 krb5kdc[6803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM
I20260925 05:01:33.932052 7393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:33.916513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52030 (local address 127.30.247.62:43329)
0925 05:01:33.916744 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:33.916751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:33.917066 (+ 315us) server_negotiation.cc:407] Connection header received
0925 05:01:33.917212 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:33.917216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:33.917272 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:33.917348 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:33.918900 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.919562 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.921370 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:33.921780 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:33.921869 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:33.926161 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:33.926168 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:33.926173 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:33.926212 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:33.929522 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:33.930141 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:33.930147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:33.930151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:33.930322 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:33.930675 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:33.930680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:33.930684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:33.931010 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:33.931273 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:33.931458 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 05:01:33.931670 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 05:01:33.935647 6846 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:01:33.936321 6846 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:01:33.943425 7263 tablet_service.cc:1467] Processing CreateTablet for tablet 8e2b91d26f414f59a1b6a70177a537b3 (DEFAULT_TABLE table=test-table [id=7b6ed1120fe14413b20cb76d263318df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:01:33.943425 6985 tablet_service.cc:1467] Processing CreateTablet for tablet 8e2b91d26f414f59a1b6a70177a537b3 (DEFAULT_TABLE table=test-table [id=7b6ed1120fe14413b20cb76d263318df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:01:33.943468 7123 tablet_service.cc:1467] Processing CreateTablet for tablet 8e2b91d26f414f59a1b6a70177a537b3 (DEFAULT_TABLE table=test-table [id=7b6ed1120fe14413b20cb76d263318df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:01:33.943852 6985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e2b91d26f414f59a1b6a70177a537b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:33.943862 7263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e2b91d26f414f59a1b6a70177a537b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:33.943913 7123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e2b91d26f414f59a1b6a70177a537b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:33.950133 7360 tablet_bootstrap.cc:492] T 8e2b91d26f414f59a1b6a70177a537b3 P 892c09a35d4248359563299977a06214: Bootstrap starting.
I20260925 05:01:33.951164 7359 tablet_bootstrap.cc:492] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399: Bootstrap starting.
I20260925 05:01:33.951756 7360 tablet_bootstrap.cc:654] T 8e2b91d26f414f59a1b6a70177a537b3 P 892c09a35d4248359563299977a06214: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:33.952379 7359 tablet_bootstrap.cc:654] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:33.952811 7358 tablet_bootstrap.cc:492] T 8e2b91d26f414f59a1b6a70177a537b3 P d8aa4ea6384a4298a04cb4b471fcad4a: Bootstrap starting.
I20260925 05:01:33.954526 7358 tablet_bootstrap.cc:654] T 8e2b91d26f414f59a1b6a70177a537b3 P d8aa4ea6384a4298a04cb4b471fcad4a: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:33.957728 7358 tablet_bootstrap.cc:492] T 8e2b91d26f414f59a1b6a70177a537b3 P d8aa4ea6384a4298a04cb4b471fcad4a: No bootstrap required, opened a new log
I20260925 05:01:33.957821 7359 tablet_bootstrap.cc:492] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399: No bootstrap required, opened a new log
I20260925 05:01:33.957885 7358 ts_tablet_manager.cc:1397] T 8e2b91d26f414f59a1b6a70177a537b3 P d8aa4ea6384a4298a04cb4b471fcad4a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260925 05:01:33.957957 7359 ts_tablet_manager.cc:1397] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260925 05:01:33.958801 7358 raft_consensus.cc:348] T 8e2b91d26f414f59a1b6a70177a537b3 P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:33.958899 7359 raft_consensus.cc:348] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:33.959097 7358 raft_consensus.cc:374] T 8e2b91d26f414f59a1b6a70177a537b3 P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:33.959180 7359 raft_consensus.cc:374] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:33.959197 7358 raft_consensus.cc:729] T 8e2b91d26f414f59a1b6a70177a537b3 P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8aa4ea6384a4298a04cb4b471fcad4a, State: Initialized, Role: FOLLOWER
I20260925 05:01:33.959278 7359 raft_consensus.cc:729] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a78bbd0e639a4d2cb96c920d629cf399, State: Initialized, Role: FOLLOWER
I20260925 05:01:33.959491 7358 consensus_queue.cc:260] T 8e2b91d26f414f59a1b6a70177a537b3 P d8aa4ea6384a4298a04cb4b471fcad4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:33.960103 7358 ts_tablet_manager.cc:1428] T 8e2b91d26f414f59a1b6a70177a537b3 P d8aa4ea6384a4298a04cb4b471fcad4a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 05:01:33.960738 7359 consensus_queue.cc:260] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:33.962193 7359 ts_tablet_manager.cc:1428] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260925 05:01:33.966008 7360 tablet_bootstrap.cc:492] T 8e2b91d26f414f59a1b6a70177a537b3 P 892c09a35d4248359563299977a06214: No bootstrap required, opened a new log
I20260925 05:01:33.966171 7360 ts_tablet_manager.cc:1397] T 8e2b91d26f414f59a1b6a70177a537b3 P 892c09a35d4248359563299977a06214: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.001s
I20260925 05:01:33.967767 7360 raft_consensus.cc:348] T 8e2b91d26f414f59a1b6a70177a537b3 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:33.967954 7360 raft_consensus.cc:374] T 8e2b91d26f414f59a1b6a70177a537b3 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:33.968066 7360 raft_consensus.cc:729] T 8e2b91d26f414f59a1b6a70177a537b3 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 892c09a35d4248359563299977a06214, State: Initialized, Role: FOLLOWER
I20260925 05:01:33.968286 7360 consensus_queue.cc:260] T 8e2b91d26f414f59a1b6a70177a537b3 P 892c09a35d4248359563299977a06214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:33.968804 7360 ts_tablet_manager.cc:1428] T 8e2b91d26f414f59a1b6a70177a537b3 P 892c09a35d4248359563299977a06214: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:01:34.087257 7366 raft_consensus.cc:482] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:01:34.087486 7366 raft_consensus.cc:504] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:34.089071 7366 leader_election.cc:288] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 892c09a35d4248359563299977a06214 (127.30.247.2:41993), d8aa4ea6384a4298a04cb4b471fcad4a (127.30.247.3:34259)
I20260925 05:01:34.096787 7405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:34.089486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:60057 (local address 127.30.247.3:34259)
0925 05:01:34.089724 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:34.089733 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:34.090582 (+ 849us) server_negotiation.cc:407] Connection header received
0925 05:01:34.090751 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:34.090755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:34.090857 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:34.090972 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:01:34.091680 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:34.092722 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:34.094479 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:34.095684 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:34.095867 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:34.095943 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:34.096297 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 05:01:34.096436 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 05:01:34.097167 7354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:34.089689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:53697 (local address 127.30.247.2:41993)
0925 05:01:34.089863 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:34.089869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:34.090247 (+ 378us) server_negotiation.cc:407] Connection header received
0925 05:01:34.090537 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:34.090543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:34.090620 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:34.090712 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:01:34.091346 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:34.092347 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:34.094877 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:34.096106 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:34.096253 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:34.096340 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:34.096747 (+ 407us) server_negotiation.cc:299] Negotiation successful
0925 05:01:34.096867 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 05:01:34.097229 7438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:34.089328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:41993 (local address 127.30.247.1:53697)
0925 05:01:34.090122 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:34.090147 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:34.090376 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:34.090770 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:34.090774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:34.090795 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:01:34.091185 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:34.091196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:34.092487 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:34.092492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:34.094708 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:34.094724 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:34.096386 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:34.096393 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:34.096579 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:34.096615 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 05:01:34.096722 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 05:01:34.096853 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":823,"threads_started":1}
I20260925 05:01:34.097782 7437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:34.089443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:34259 (local address 127.30.247.1:60057)
0925 05:01:34.090475 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:34.090493 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:34.090622 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:34.091009 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:34.091014 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:34.091044 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:01:34.091483 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:34.091494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:34.092845 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:34.092852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:34.094308 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:34.094322 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:34.095921 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:34.095926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:34.096105 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:34.096141 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 05:01:34.097026 (+ 885us) client_negotiation.cc:240] Negotiation successful
0925 05:01:34.097207 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":904,"mutex_wait_us":7,"thread_start_us":408,"threads_started":1}
I20260925 05:01:34.097949 7148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2b91d26f414f59a1b6a70177a537b3" candidate_uuid: "a78bbd0e639a4d2cb96c920d629cf399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "892c09a35d4248359563299977a06214" is_pre_election: true
I20260925 05:01:34.098217 7148 raft_consensus.cc:2454] T 8e2b91d26f414f59a1b6a70177a537b3 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a78bbd0e639a4d2cb96c920d629cf399 in term 0.
I20260925 05:01:34.098491 7288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2b91d26f414f59a1b6a70177a537b3" candidate_uuid: "a78bbd0e639a4d2cb96c920d629cf399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" is_pre_election: true
I20260925 05:01:34.098709 7288 raft_consensus.cc:2454] T 8e2b91d26f414f59a1b6a70177a537b3 P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a78bbd0e639a4d2cb96c920d629cf399 in term 0.
I20260925 05:01:34.098718 6937 leader_election.cc:302] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 892c09a35d4248359563299977a06214, a78bbd0e639a4d2cb96c920d629cf399; no voters:
I20260925 05:01:34.099120 7366 raft_consensus.cc:2781] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:01:34.099254 7366 raft_consensus.cc:482] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:01:34.099360 7366 raft_consensus.cc:3037] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:34.101779 7366 raft_consensus.cc:504] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:34.102355 7366 leader_election.cc:288] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399 [CANDIDATE]: Term 1 election: Requested vote from peers 892c09a35d4248359563299977a06214 (127.30.247.2:41993), d8aa4ea6384a4298a04cb4b471fcad4a (127.30.247.3:34259)
I20260925 05:01:34.102782 7148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2b91d26f414f59a1b6a70177a537b3" candidate_uuid: "a78bbd0e639a4d2cb96c920d629cf399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "892c09a35d4248359563299977a06214"
I20260925 05:01:34.102859 7288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2b91d26f414f59a1b6a70177a537b3" candidate_uuid: "a78bbd0e639a4d2cb96c920d629cf399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a"
I20260925 05:01:34.102983 7148 raft_consensus.cc:3037] T 8e2b91d26f414f59a1b6a70177a537b3 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:34.103056 7288 raft_consensus.cc:3037] T 8e2b91d26f414f59a1b6a70177a537b3 P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:34.105957 7148 raft_consensus.cc:2454] T 8e2b91d26f414f59a1b6a70177a537b3 P 892c09a35d4248359563299977a06214 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a78bbd0e639a4d2cb96c920d629cf399 in term 1.
I20260925 05:01:34.106336 6937 leader_election.cc:302] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 892c09a35d4248359563299977a06214, a78bbd0e639a4d2cb96c920d629cf399; no voters:
I20260925 05:01:34.106485 7288 raft_consensus.cc:2454] T 8e2b91d26f414f59a1b6a70177a537b3 P d8aa4ea6384a4298a04cb4b471fcad4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a78bbd0e639a4d2cb96c920d629cf399 in term 1.
I20260925 05:01:34.106642 7366 raft_consensus.cc:2781] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:01:34.107306 7366 raft_consensus.cc:686] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399 [term 1 LEADER]: Becoming Leader. State: Replica: a78bbd0e639a4d2cb96c920d629cf399, State: Running, Role: LEADER
I20260925 05:01:34.107659 7366 consensus_queue.cc:237] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:34.114900 6847 catalog_manager.cc:5495] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399 reported cstate change: term changed from 0 to 1, leader changed from <none> to a78bbd0e639a4d2cb96c920d629cf399 (127.30.247.1). New cstate: current_term: 1 leader_uuid: "a78bbd0e639a4d2cb96c920d629cf399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } health_report { overall_health: UNKNOWN } } }
I20260925 05:01:34.154315 7442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:34.148692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34086 (local address 127.30.247.1:36451)
0925 05:01:34.149142 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:34.149146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:34.149162 (+ 16us) server_negotiation.cc:407] Connection header received
0925 05:01:34.149295 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:34.149298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:34.149376 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:34.149508 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:01:34.149847 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:34.150596 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:34.152162 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:34.152591 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:34.152683 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:34.153113 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:01:34.153643 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:01:34.153745 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:34.153928 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 05:01:34.153987 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":267,"threads_started":1}
I20260925 05:01:34.157877 7148 raft_consensus.cc:1261] T 8e2b91d26f414f59a1b6a70177a537b3 P 892c09a35d4248359563299977a06214 [term 1 FOLLOWER]: Refusing update from remote peer a78bbd0e639a4d2cb96c920d629cf399: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:01:34.158115 7288 raft_consensus.cc:1261] T 8e2b91d26f414f59a1b6a70177a537b3 P d8aa4ea6384a4298a04cb4b471fcad4a [term 1 FOLLOWER]: Refusing update from remote peer a78bbd0e639a4d2cb96c920d629cf399: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:01:34.158623 7439 consensus_queue.cc:1035] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:01:34.158983 7366 consensus_queue.cc:1035] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:01:34.178130 6847 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:01:34.178349 6847 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:01:34.181191 6847 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32cd628ad4834acdba076e9b198ed8dc: Sending DeleteTablet for 3 replicas of tablet 8e2b91d26f414f59a1b6a70177a537b3
I20260925 05:01:34.181874 6985 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e2b91d26f414f59a1b6a70177a537b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:01:34 UTC) from {username='oryx', principal='oryx/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:34078
I20260925 05:01:34.181895 7123 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e2b91d26f414f59a1b6a70177a537b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:01:34 UTC) from {username='oryx', principal='oryx/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:45224
I20260925 05:01:34.182120 7419 tablet_replica.cc:330] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399: stopping tablet replica
I20260925 05:01:34.182132 7263 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e2b91d26f414f59a1b6a70177a537b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:01:34 UTC) from {username='oryx', principal='oryx/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:58220
I20260925 05:01:34.182307 7419 raft_consensus.cc:2229] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:01:34.182363 7421 tablet_replica.cc:330] T 8e2b91d26f414f59a1b6a70177a537b3 P d8aa4ea6384a4298a04cb4b471fcad4a: stopping tablet replica
I20260925 05:01:34.182538 7421 raft_consensus.cc:2229] T 8e2b91d26f414f59a1b6a70177a537b3 P d8aa4ea6384a4298a04cb4b471fcad4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:01:34.182717 7421 raft_consensus.cc:2258] T 8e2b91d26f414f59a1b6a70177a537b3 P d8aa4ea6384a4298a04cb4b471fcad4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:01:34.182715 7419 raft_consensus.cc:2258] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:01:34.182971 7420 tablet_replica.cc:330] T 8e2b91d26f414f59a1b6a70177a537b3 P 892c09a35d4248359563299977a06214: stopping tablet replica
I20260925 05:01:34.183157 7420 raft_consensus.cc:2229] T 8e2b91d26f414f59a1b6a70177a537b3 P 892c09a35d4248359563299977a06214 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:01:34.183353 7420 raft_consensus.cc:2258] T 8e2b91d26f414f59a1b6a70177a537b3 P 892c09a35d4248359563299977a06214 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:01:34.183693 7421 ts_tablet_manager.cc:1905] T 8e2b91d26f414f59a1b6a70177a537b3 P d8aa4ea6384a4298a04cb4b471fcad4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:01:34.184012 7419 ts_tablet_manager.cc:1905] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:01:34.184695 7420 ts_tablet_manager.cc:1905] T 8e2b91d26f414f59a1b6a70177a537b3 P 892c09a35d4248359563299977a06214: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:01:34.190666 7419 ts_tablet_manager.cc:1918] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:01:34.190790 7421 ts_tablet_manager.cc:1918] T 8e2b91d26f414f59a1b6a70177a537b3 P d8aa4ea6384a4298a04cb4b471fcad4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:01:34.190831 7419 log.cc:1196] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-0/wal/wals/8e2b91d26f414f59a1b6a70177a537b3
I20260925 05:01:34.190932 7421 log.cc:1196] T 8e2b91d26f414f59a1b6a70177a537b3 P d8aa4ea6384a4298a04cb4b471fcad4a: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-2/wal/wals/8e2b91d26f414f59a1b6a70177a537b3
I20260925 05:01:34.191258 7419 ts_tablet_manager.cc:1939] T 8e2b91d26f414f59a1b6a70177a537b3 P a78bbd0e639a4d2cb96c920d629cf399: Deleting consensus metadata
I20260925 05:01:34.191322 7421 ts_tablet_manager.cc:1939] T 8e2b91d26f414f59a1b6a70177a537b3 P d8aa4ea6384a4298a04cb4b471fcad4a: Deleting consensus metadata
I20260925 05:01:34.191707 7420 ts_tablet_manager.cc:1918] T 8e2b91d26f414f59a1b6a70177a537b3 P 892c09a35d4248359563299977a06214: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:01:34.191912 7420 log.cc:1196] T 8e2b91d26f414f59a1b6a70177a537b3 P 892c09a35d4248359563299977a06214: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-1/wal/wals/8e2b91d26f414f59a1b6a70177a537b3
I20260925 05:01:34.192281 6832 catalog_manager.cc:4841] TS d8aa4ea6384a4298a04cb4b471fcad4a (127.30.247.3:34259): tablet 8e2b91d26f414f59a1b6a70177a537b3 (table test-table [id=7b6ed1120fe14413b20cb76d263318df]) successfully deleted
I20260925 05:01:34.192333 6835 catalog_manager.cc:4841] TS a78bbd0e639a4d2cb96c920d629cf399 (127.30.247.1:36451): tablet 8e2b91d26f414f59a1b6a70177a537b3 (table test-table [id=7b6ed1120fe14413b20cb76d263318df]) successfully deleted
I20260925 05:01:34.192503 6847 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 05:01:34.192924 7420 ts_tablet_manager.cc:1939] T 8e2b91d26f414f59a1b6a70177a537b3 P 892c09a35d4248359563299977a06214: Deleting consensus metadata
W20260925 05:01:34.193207 6847 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:01:34.193964 6834 catalog_manager.cc:4841] TS 892c09a35d4248359563299977a06214 (127.30.247.2:41993): tablet 8e2b91d26f414f59a1b6a70177a537b3 (table test-table [id=7b6ed1120fe14413b20cb76d263318df]) successfully deleted
I20260925 05:01:34.199743 6985 tablet_service.cc:1467] Processing CreateTablet for tablet d404a1fa034244a588f903a0048af5d3 (DEFAULT_TABLE table=test-table [id=928bae51c00042b0ba1bd93106d58b37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:01:34.200167 6985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d404a1fa034244a588f903a0048af5d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:34.200300 7263 tablet_service.cc:1467] Processing CreateTablet for tablet d404a1fa034244a588f903a0048af5d3 (DEFAULT_TABLE table=test-table [id=928bae51c00042b0ba1bd93106d58b37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:01:34.200762 7263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d404a1fa034244a588f903a0048af5d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:34.206256 7358 tablet_bootstrap.cc:492] T d404a1fa034244a588f903a0048af5d3 P d8aa4ea6384a4298a04cb4b471fcad4a: Bootstrap starting.
I20260925 05:01:34.206876 7359 tablet_bootstrap.cc:492] T d404a1fa034244a588f903a0048af5d3 P a78bbd0e639a4d2cb96c920d629cf399: Bootstrap starting.
I20260925 05:01:34.207201 7123 tablet_service.cc:1467] Processing CreateTablet for tablet d404a1fa034244a588f903a0048af5d3 (DEFAULT_TABLE table=test-table [id=928bae51c00042b0ba1bd93106d58b37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:01:34.207609 7123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d404a1fa034244a588f903a0048af5d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:34.207957 7358 tablet_bootstrap.cc:654] T d404a1fa034244a588f903a0048af5d3 P d8aa4ea6384a4298a04cb4b471fcad4a: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:34.208467 7359 tablet_bootstrap.cc:654] T d404a1fa034244a588f903a0048af5d3 P a78bbd0e639a4d2cb96c920d629cf399: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:34.216432 7360 tablet_bootstrap.cc:492] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214: Bootstrap starting.
I20260925 05:01:34.218665 7360 tablet_bootstrap.cc:654] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:34.219902 7359 tablet_bootstrap.cc:492] T d404a1fa034244a588f903a0048af5d3 P a78bbd0e639a4d2cb96c920d629cf399: No bootstrap required, opened a new log
I20260925 05:01:34.220124 7359 ts_tablet_manager.cc:1397] T d404a1fa034244a588f903a0048af5d3 P a78bbd0e639a4d2cb96c920d629cf399: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.002s
I20260925 05:01:34.220927 7359 raft_consensus.cc:348] T d404a1fa034244a588f903a0048af5d3 P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:34.221251 7359 raft_consensus.cc:374] T d404a1fa034244a588f903a0048af5d3 P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:34.221381 7359 raft_consensus.cc:729] T d404a1fa034244a588f903a0048af5d3 P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a78bbd0e639a4d2cb96c920d629cf399, State: Initialized, Role: FOLLOWER
I20260925 05:01:34.221696 7359 consensus_queue.cc:260] T d404a1fa034244a588f903a0048af5d3 P a78bbd0e639a4d2cb96c920d629cf399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:34.222621 7360 tablet_bootstrap.cc:492] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214: No bootstrap required, opened a new log
I20260925 05:01:34.222805 7360 ts_tablet_manager.cc:1397] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.003s
I20260925 05:01:34.223367 7360 raft_consensus.cc:348] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:34.223591 7360 raft_consensus.cc:374] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:34.223698 7360 raft_consensus.cc:729] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 892c09a35d4248359563299977a06214, State: Initialized, Role: FOLLOWER
I20260925 05:01:34.223939 7360 consensus_queue.cc:260] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:34.224429 7360 ts_tablet_manager.cc:1428] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 05:01:34.225353 7358 tablet_bootstrap.cc:492] T d404a1fa034244a588f903a0048af5d3 P d8aa4ea6384a4298a04cb4b471fcad4a: No bootstrap required, opened a new log
I20260925 05:01:34.225510 7358 ts_tablet_manager.cc:1397] T d404a1fa034244a588f903a0048af5d3 P d8aa4ea6384a4298a04cb4b471fcad4a: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.002s
I20260925 05:01:34.226224 7359 ts_tablet_manager.cc:1428] T d404a1fa034244a588f903a0048af5d3 P a78bbd0e639a4d2cb96c920d629cf399: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260925 05:01:34.226137 7358 raft_consensus.cc:348] T d404a1fa034244a588f903a0048af5d3 P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:34.226362 7358 raft_consensus.cc:374] T d404a1fa034244a588f903a0048af5d3 P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:34.226435 7358 raft_consensus.cc:729] T d404a1fa034244a588f903a0048af5d3 P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8aa4ea6384a4298a04cb4b471fcad4a, State: Initialized, Role: FOLLOWER
I20260925 05:01:34.226683 7358 consensus_queue.cc:260] T d404a1fa034244a588f903a0048af5d3 P d8aa4ea6384a4298a04cb4b471fcad4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:34.227874 7358 ts_tablet_manager.cc:1428] T d404a1fa034244a588f903a0048af5d3 P d8aa4ea6384a4298a04cb4b471fcad4a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 05:01:34.441515 7365 raft_consensus.cc:482] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:01:34.441733 7365 raft_consensus.cc:504] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:34.442516 7365 leader_election.cc:288] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a78bbd0e639a4d2cb96c920d629cf399 (127.30.247.1:36451), d8aa4ea6384a4298a04cb4b471fcad4a (127.30.247.3:34259)
I20260925 05:01:34.443035 7288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d404a1fa034244a588f903a0048af5d3" candidate_uuid: "892c09a35d4248359563299977a06214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" is_pre_election: true
I20260925 05:01:34.443014 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d404a1fa034244a588f903a0048af5d3" candidate_uuid: "892c09a35d4248359563299977a06214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a78bbd0e639a4d2cb96c920d629cf399" is_pre_election: true
I20260925 05:01:34.443276 7288 raft_consensus.cc:2454] T d404a1fa034244a588f903a0048af5d3 P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 892c09a35d4248359563299977a06214 in term 0.
I20260925 05:01:34.443322 7010 raft_consensus.cc:2454] T d404a1fa034244a588f903a0048af5d3 P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 892c09a35d4248359563299977a06214 in term 0.
I20260925 05:01:34.443760 7075 leader_election.cc:302] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 892c09a35d4248359563299977a06214, d8aa4ea6384a4298a04cb4b471fcad4a; no voters:
I20260925 05:01:34.444056 7365 raft_consensus.cc:2781] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:01:34.444181 7365 raft_consensus.cc:482] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:01:34.444306 7365 raft_consensus.cc:3037] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:34.446295 7365 raft_consensus.cc:504] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:34.446836 7365 leader_election.cc:288] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214 [CANDIDATE]: Term 1 election: Requested vote from peers a78bbd0e639a4d2cb96c920d629cf399 (127.30.247.1:36451), d8aa4ea6384a4298a04cb4b471fcad4a (127.30.247.3:34259)
I20260925 05:01:34.447220 7010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d404a1fa034244a588f903a0048af5d3" candidate_uuid: "892c09a35d4248359563299977a06214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a78bbd0e639a4d2cb96c920d629cf399"
I20260925 05:01:34.447371 7288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d404a1fa034244a588f903a0048af5d3" candidate_uuid: "892c09a35d4248359563299977a06214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a"
I20260925 05:01:34.447434 7010 raft_consensus.cc:3037] T d404a1fa034244a588f903a0048af5d3 P a78bbd0e639a4d2cb96c920d629cf399 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:34.447556 7288 raft_consensus.cc:3037] T d404a1fa034244a588f903a0048af5d3 P d8aa4ea6384a4298a04cb4b471fcad4a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:34.449831 7010 raft_consensus.cc:2454] T d404a1fa034244a588f903a0048af5d3 P a78bbd0e639a4d2cb96c920d629cf399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 892c09a35d4248359563299977a06214 in term 1.
I20260925 05:01:34.450218 7078 leader_election.cc:302] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 892c09a35d4248359563299977a06214, a78bbd0e639a4d2cb96c920d629cf399; no voters:
I20260925 05:01:34.450444 7288 raft_consensus.cc:2454] T d404a1fa034244a588f903a0048af5d3 P d8aa4ea6384a4298a04cb4b471fcad4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 892c09a35d4248359563299977a06214 in term 1.
I20260925 05:01:34.450477 7365 raft_consensus.cc:2781] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:01:34.452730 7365 raft_consensus.cc:686] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214 [term 1 LEADER]: Becoming Leader. State: Replica: 892c09a35d4248359563299977a06214, State: Running, Role: LEADER
I20260925 05:01:34.453037 7365 consensus_queue.cc:237] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } }
I20260925 05:01:34.455654 6847 catalog_manager.cc:5495] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214 reported cstate change: term changed from 0 to 1, leader changed from <none> to 892c09a35d4248359563299977a06214 (127.30.247.2). New cstate: current_term: 1 leader_uuid: "892c09a35d4248359563299977a06214" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "892c09a35d4248359563299977a06214" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 41993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 } health_report { overall_health: UNKNOWN } } }
I20260925 05:01:34.512921 7354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:34.507361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45248 (local address 127.30.247.2:41993)
0925 05:01:34.507536 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:34.507541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:34.507558 (+ 17us) server_negotiation.cc:407] Connection header received
0925 05:01:34.507825 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:34.507829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:34.507890 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:34.507967 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:01:34.508408 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:34.509235 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:34.511132 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:34.511503 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:34.511589 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:34.512143 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:01:34.512246 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:01:34.512323 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:34.512496 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 05:01:34.512625 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 05:01:34.522868 7288 raft_consensus.cc:1261] T d404a1fa034244a588f903a0048af5d3 P d8aa4ea6384a4298a04cb4b471fcad4a [term 1 FOLLOWER]: Refusing update from remote peer 892c09a35d4248359563299977a06214: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:01:34.523155 7010 raft_consensus.cc:1261] T d404a1fa034244a588f903a0048af5d3 P a78bbd0e639a4d2cb96c920d629cf399 [term 1 FOLLOWER]: Refusing update from remote peer 892c09a35d4248359563299977a06214: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:01:34.523768 7365 consensus_queue.cc:1035] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8aa4ea6384a4298a04cb4b471fcad4a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 34259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:01:34.524071 7450 consensus_queue.cc:1035] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a78bbd0e639a4d2cb96c920d629cf399" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:01:34.578956 6847 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:01:34.579210 6847 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:01:34.581964 6847 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32cd628ad4834acdba076e9b198ed8dc: Sending DeleteTablet for 3 replicas of tablet d404a1fa034244a588f903a0048af5d3
I20260925 05:01:34.582669 6985 tablet_service.cc:1514] Processing DeleteTablet for tablet d404a1fa034244a588f903a0048af5d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:01:34 UTC) from {username='oryx', principal='oryx/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:34078
I20260925 05:01:34.582716 7263 tablet_service.cc:1514] Processing DeleteTablet for tablet d404a1fa034244a588f903a0048af5d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:01:34 UTC) from {username='oryx', principal='oryx/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:58220
I20260925 05:01:34.583006 7123 tablet_service.cc:1514] Processing DeleteTablet for tablet d404a1fa034244a588f903a0048af5d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:01:34 UTC) from {username='oryx', principal='oryx/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:45224
I20260925 05:01:34.583163 7421 tablet_replica.cc:330] T d404a1fa034244a588f903a0048af5d3 P d8aa4ea6384a4298a04cb4b471fcad4a: stopping tablet replica
I20260925 05:01:34.583263 7420 tablet_replica.cc:330] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214: stopping tablet replica
I20260925 05:01:34.583330 7421 raft_consensus.cc:2229] T d404a1fa034244a588f903a0048af5d3 P d8aa4ea6384a4298a04cb4b471fcad4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:01:34.583423 7420 raft_consensus.cc:2229] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:01:34.583499 7421 raft_consensus.cc:2258] T d404a1fa034244a588f903a0048af5d3 P d8aa4ea6384a4298a04cb4b471fcad4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:01:34.583719 7420 raft_consensus.cc:2258] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:01:34.583746 7419 tablet_replica.cc:330] T d404a1fa034244a588f903a0048af5d3 P a78bbd0e639a4d2cb96c920d629cf399: stopping tablet replica
I20260925 05:01:34.583904 7419 raft_consensus.cc:2229] T d404a1fa034244a588f903a0048af5d3 P a78bbd0e639a4d2cb96c920d629cf399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:01:34.584016 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 6919
I20260925 05:01:34.584088 7419 raft_consensus.cc:2258] T d404a1fa034244a588f903a0048af5d3 P a78bbd0e639a4d2cb96c920d629cf399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:01:34.585310 7419 ts_tablet_manager.cc:1905] T d404a1fa034244a588f903a0048af5d3 P a78bbd0e639a4d2cb96c920d629cf399: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:01:34.585345 7421 ts_tablet_manager.cc:1905] T d404a1fa034244a588f903a0048af5d3 P d8aa4ea6384a4298a04cb4b471fcad4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:01:34.589661 7420 ts_tablet_manager.cc:1905] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:01:34.591701 7421 ts_tablet_manager.cc:1918] T d404a1fa034244a588f903a0048af5d3 P d8aa4ea6384a4298a04cb4b471fcad4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:01:34.591813 7421 log.cc:1196] T d404a1fa034244a588f903a0048af5d3 P d8aa4ea6384a4298a04cb4b471fcad4a: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-2/wal/wals/d404a1fa034244a588f903a0048af5d3
I20260925 05:01:34.592188 7421 ts_tablet_manager.cc:1939] T d404a1fa034244a588f903a0048af5d3 P d8aa4ea6384a4298a04cb4b471fcad4a: Deleting consensus metadata
I20260925 05:01:34.593187 6832 catalog_manager.cc:4841] TS d8aa4ea6384a4298a04cb4b471fcad4a (127.30.247.3:34259): tablet d404a1fa034244a588f903a0048af5d3 (table test-table [id=928bae51c00042b0ba1bd93106d58b37]) successfully deleted
I20260925 05:01:34.597622 7419 ts_tablet_manager.cc:1918] T d404a1fa034244a588f903a0048af5d3 P a78bbd0e639a4d2cb96c920d629cf399: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:01:34.597874 7419 log.cc:1196] T d404a1fa034244a588f903a0048af5d3 P a78bbd0e639a4d2cb96c920d629cf399: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-0/wal/wals/d404a1fa034244a588f903a0048af5d3
I20260925 05:01:34.598273 7420 ts_tablet_manager.cc:1918] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:01:34.598414 7419 ts_tablet_manager.cc:1939] T d404a1fa034244a588f903a0048af5d3 P a78bbd0e639a4d2cb96c920d629cf399: Deleting consensus metadata
I20260925 05:01:34.598640 7420 log.cc:1196] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312430481417-31708-0/minicluster-data/ts-1/wal/wals/d404a1fa034244a588f903a0048af5d3
I20260925 05:01:34.599056 7420 ts_tablet_manager.cc:1939] T d404a1fa034244a588f903a0048af5d3 P 892c09a35d4248359563299977a06214: Deleting consensus metadata
I20260925 05:01:34.599570 6835 catalog_manager.cc:4841] TS a78bbd0e639a4d2cb96c920d629cf399 (127.30.247.1:36451): tablet d404a1fa034244a588f903a0048af5d3 (table test-table [id=928bae51c00042b0ba1bd93106d58b37]) successfully deleted
I20260925 05:01:34.600157 6834 catalog_manager.cc:4841] TS 892c09a35d4248359563299977a06214 (127.30.247.2:41993): tablet d404a1fa034244a588f903a0048af5d3 (table test-table [id=928bae51c00042b0ba1bd93106d58b37]) successfully deleted
Sep 25 05:01:34 dist-test-slave-5n86 krb5kdc[6803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.247.1@KRBTEST.COM
I20260925 05:01:34.611487 7442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:34.585951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34100 (local address 127.30.247.1:36451)
0925 05:01:34.588709 (+ 2758us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:34.588723 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:34.593920 (+ 5197us) server_negotiation.cc:407] Connection header received
0925 05:01:34.594073 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:34.594080 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:34.594157 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:34.594254 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:34.595869 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:34.597054 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:34.599534 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:34.600201 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:34.600318 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:34.604970 (+ 4652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:34.604983 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:34.605003 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:34.605041 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:34.608956 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:34.609613 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:34.609622 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:34.609625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:34.609771 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:34.610096 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:34.610101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:34.610105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:34.610489 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:34.610795 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:34.610935 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 05:01:34.611144 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2601}
I20260925 05:01:34.612176 7050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:34.742640 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 6919
I20260925 05:01:34.765424 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 7059
Sep 25 05:01:34 dist-test-slave-5n86 krb5kdc[6803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.247.2@KRBTEST.COM
I20260925 05:01:34.779964 7354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:34.766171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45256 (local address 127.30.247.2:41993)
0925 05:01:34.766381 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:34.766387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:34.766505 (+ 118us) server_negotiation.cc:407] Connection header received
0925 05:01:34.766734 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:34.766739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:34.766816 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:34.766939 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:34.768151 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:34.769003 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:34.770213 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:34.770607 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:34.770744 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:34.774871 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:34.774884 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:34.774898 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:34.774936 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:34.777489 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:34.778099 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:34.778111 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:34.778113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:34.778239 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:34.778654 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:34.778660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:34.778676 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:34.779070 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:34.779314 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:34.779540 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 05:01:34.779696 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 05:01:34.780714 7189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:34.902637 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 7059
I20260925 05:01:34.924499 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 7198
Sep 25 05:01:34 dist-test-slave-5n86 krb5kdc[6803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.247.3@KRBTEST.COM
I20260925 05:01:34.939060 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:34.925249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58248 (local address 127.30.247.3:34259)
0925 05:01:34.925727 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:34.925735 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:34.925755 (+ 20us) server_negotiation.cc:407] Connection header received
0925 05:01:34.925804 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:34.925807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:34.925861 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:34.925975 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:34.927237 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:34.927931 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:34.929305 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:34.929758 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:34.929884 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:34.934000 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:34.934013 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:34.934028 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:34.934063 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:34.936627 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:34.937466 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:34.937473 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:34.937476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:34.937594 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:34.937889 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:34.937892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:34.937894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:34.938230 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:34.938442 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:34.938670 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 05:01:34.938816 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":240,"threads_started":1}
I20260925 05:01:34.939728 7328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:35.058076 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 7198
I20260925 05:01:35.074965 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 6816
I20260925 05:01:35.076215 6888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:35.187954 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 6816
2026-09-25T05:01:35Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6780 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:01:35 dist-test-slave-5n86 krb5kdc[7469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:01:35 dist-test-slave-5n86 krb5kdc[7469](info): set up 2 sockets
Sep 25 05:01:35 dist-test-slave-5n86 krb5kdc[7469](info): commencing operation
krb5kdc: starting...
W20260925 05:01:36.277789 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:01:36 dist-test-slave-5n86 krb5kdc[7469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:01:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:01:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:01:36.476625 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:34025
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:40651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:34025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:36.784255 7482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:36.784512 7482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:36.784706 7482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:36.791541 7482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:01:36.791654 7482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:36.791734 7482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:36.791807 7482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:01:36.791877 7482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:01:36.801046 7482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:40651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:34025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:34025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:36.802179 7482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:36.803417 7482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:36.814425 7487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:36.814459 7488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:36.814621 7490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:36.815730 7482 server_base.cc:1034] running on GCE node
I20260925 05:01:36.816357 7482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:36.817627 7482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:36.818949 7482 hybrid_clock.cc:648] HybridClock initialized: now 1790312496818782 us; error 42 us; skew 500 ppm
I20260925 05:01:36.819254 7482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T05:01:36Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1703 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:01:36 dist-test-slave-5n86 krb5kdc[7496](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:01:36 dist-test-slave-5n86 krb5kdc[7496](info): set up 2 sockets
Sep 25 05:01:36 dist-test-slave-5n86 krb5kdc[7496](info): commencing operation
krb5kdc: starting...
W20260925 05:01:37.981715 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:01:38 dist-test-slave-5n86 krb5kdc[7496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:01:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:01:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:01:38.179615 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:37869
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:37869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:38.486127 7509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:38.486389 7509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:38.486477 7509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:38.493950 7509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:01:38.494068 7509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:38.494122 7509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:38.494177 7509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:01:38.494220 7509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:01:38.503597 7509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:37869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:37869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:38.504899 7509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:38.506124 7509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:38.520733 7514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:38.520802 7517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:38.520776 7515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:38.521661 7509 server_base.cc:1034] running on GCE node
I20260925 05:01:38.522382 7509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:38.523767 7509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:38.525012 7509 hybrid_clock.cc:648] HybridClock initialized: now 1790312498524945 us; error 58 us; skew 500 ppm
I20260925 05:01:38.525408 7509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:38 dist-test-slave-5n86 krb5kdc[7496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312498, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:38.531664 7509 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:01:38.533104 7509 webserver.cc:460] Webserver started at http://127.30.247.62:41137/ using document root <none> and password file <none>
I20260925 05:01:38.533764 7509 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:38.533902 7509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:38.534197 7509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:38.537025 7509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "12a5806c85304822895da797c83d4a11"
format_stamp: "Formatted at 2026-09-25 05:01:38 on dist-test-slave-5n86"
server_key: "6f6388068cdf2a8923b6516c7f3f0fd9"
server_key_iv: "6f09723b8a2a97ab6bb5eb645b57c70d"
server_key_version: "encryptionkey@0"
I20260925 05:01:38.537742 7509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "12a5806c85304822895da797c83d4a11"
format_stamp: "Formatted at 2026-09-25 05:01:38 on dist-test-slave-5n86"
server_key: "6f6388068cdf2a8923b6516c7f3f0fd9"
server_key_iv: "6f09723b8a2a97ab6bb5eb645b57c70d"
server_key_version: "encryptionkey@0"
I20260925 05:01:38.542416 7509 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 05:01:38.545539 7524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:38.546770 7509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:01:38.546936 7509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "12a5806c85304822895da797c83d4a11"
format_stamp: "Formatted at 2026-09-25 05:01:38 on dist-test-slave-5n86"
server_key: "6f6388068cdf2a8923b6516c7f3f0fd9"
server_key_iv: "6f09723b8a2a97ab6bb5eb645b57c70d"
server_key_version: "encryptionkey@0"
I20260925 05:01:38.547113 7509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:38.588680 7509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:38.592288 7509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:38.592582 7509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:38.610446 7509 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:37869
I20260925 05:01:38.610476 7575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:37869 every 8 connection(s)
I20260925 05:01:38.611675 7509 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:01:38.615069 7576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:38.616703 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 7509
I20260925 05:01:38.616919 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:01:38.617736 31708 external_mini_cluster.cc:1428] Setting key 4549a22ca6f500a3099c7b46551525f3
I20260925 05:01:38.629127 7576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11: Bootstrap starting.
Sep 25 05:01:38 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:01:38.632892 7576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:38.633899 7576 log.cc:824] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11: Log is configured to *not* fsync() on all Append() calls
I20260925 05:01:38.637274 7576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11: No bootstrap required, opened a new log
I20260925 05:01:38.641345 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:38.619728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47592 (local address 127.30.247.62:37869)
0925 05:01:38.620570 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:38.620587 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:38.620637 (+ 50us) server_negotiation.cc:407] Connection header received
0925 05:01:38.622203 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:38.622227 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:38.622594 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:38.622885 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:38.624770 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:38.626097 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:38.627896 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:38.628597 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:38.628740 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:38.633324 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:38.633331 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:38.633344 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:38.633377 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:38.637130 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:38.637782 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:38.637790 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:38.637798 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:38.638000 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:38.638336 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:38.638340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:38.638344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:38.638989 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:38.639286 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:38.639590 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 05:01:38.640021 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":393,"threads_started":1}
I20260925 05:01:38.645825 7576 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a5806c85304822895da797c83d4a11" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37869 } }
I20260925 05:01:38.646130 7576 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:38.646203 7576 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a5806c85304822895da797c83d4a11, State: Initialized, Role: FOLLOWER
I20260925 05:01:38.646644 7576 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a5806c85304822895da797c83d4a11" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37869 } }
I20260925 05:01:38.646831 7576 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:01:38.646917 7576 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:01:38.647032 7576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:38.649047 7576 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a5806c85304822895da797c83d4a11" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37869 } }
I20260925 05:01:38.649446 7576 leader_election.cc:302] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12a5806c85304822895da797c83d4a11; no voters:
I20260925 05:01:38.650156 7576 leader_election.cc:288] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:01:38.650249 7581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:01:38.650849 7581 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [term 1 LEADER]: Becoming Leader. State: Replica: 12a5806c85304822895da797c83d4a11, State: Running, Role: LEADER
I20260925 05:01:38.651427 7581 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a5806c85304822895da797c83d4a11" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37869 } }
I20260925 05:01:38.652351 7576 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:01:38.654839 7582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12a5806c85304822895da797c83d4a11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a5806c85304822895da797c83d4a11" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37869 } } }
I20260925 05:01:38.655584 7582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:38.654830 7583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12a5806c85304822895da797c83d4a11. Latest consensus state: current_term: 1 leader_uuid: "12a5806c85304822895da797c83d4a11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a5806c85304822895da797c83d4a11" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37869 } } }
I20260925 05:01:38.655911 7583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:38.657948 7586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:01:38.662309 7586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:01:38.670252 7586 catalog_manager.cc:1347] Generated new cluster ID: 8ee05980f31b45c38ae9da6de062ce65
I20260925 05:01:38.670354 7586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:01:38.689121 7586 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:01:38.690773 7586 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:01:38.706168 7586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11: Generated new TSK 0
I20260925 05:01:38.706890 7586 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
I20260925 05:01:38.764885 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:0
--local_ip_for_outbound_sockets=127.30.247.1
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:37869
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:39.078390 7602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:39.078646 7602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:39.078734 7602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:39.085556 7602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:39.085646 7602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:39.085749 7602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:01:39.094681 7602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:37869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:39.095760 7602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:39.096946 7602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:39.109407 7610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:39.109524 7607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:39.109414 7608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:39.110405 7602 server_base.cc:1034] running on GCE node
I20260925 05:01:39.110939 7602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:39.111826 7602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:39.113037 7602 hybrid_clock.cc:648] HybridClock initialized: now 1790312499113006 us; error 52 us; skew 500 ppm
I20260925 05:01:39.113337 7602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:39 dist-test-slave-5n86 krb5kdc[7496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312499, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:39.116326 7602 init.cc:377] Logged in from keytab as kudu/127.30.247.1@KRBTEST.COM (short username kudu)
I20260925 05:01:39.117703 7602 webserver.cc:460] Webserver started at http://127.30.247.1:43295/ using document root <none> and password file <none>
I20260925 05:01:39.118346 7602 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:39.118487 7602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:39.118775 7602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:39.121507 7602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/data/instance:
uuid: "bedce651edc54e248c584b27e01e8dd8"
format_stamp: "Formatted at 2026-09-25 05:01:39 on dist-test-slave-5n86"
server_key: "37c81caae811f4d19eaf2158644dc7d9"
server_key_iv: "31bdc6b3d3d75b459993aad259543ff1"
server_key_version: "encryptionkey@0"
I20260925 05:01:39.122298 7602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance:
uuid: "bedce651edc54e248c584b27e01e8dd8"
format_stamp: "Formatted at 2026-09-25 05:01:39 on dist-test-slave-5n86"
server_key: "37c81caae811f4d19eaf2158644dc7d9"
server_key_iv: "31bdc6b3d3d75b459993aad259543ff1"
server_key_version: "encryptionkey@0"
I20260925 05:01:39.127095 7602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 05:01:39.130288 7617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:39.131522 7602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:01:39.131690 7602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "bedce651edc54e248c584b27e01e8dd8"
format_stamp: "Formatted at 2026-09-25 05:01:39 on dist-test-slave-5n86"
server_key: "37c81caae811f4d19eaf2158644dc7d9"
server_key_iv: "31bdc6b3d3d75b459993aad259543ff1"
server_key_version: "encryptionkey@0"
I20260925 05:01:39.131870 7602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:39.142865 7602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:39.146768 7602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:39.147042 7602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:39.147886 7602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:39.149178 7602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:39.149276 7602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:39.149416 7602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:39.149487 7602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:39.181948 7602 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:38189
I20260925 05:01:39.181998 7729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:38189 every 8 connection(s)
I20260925 05:01:39.183175 7602 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:01:39.193076 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 7602
I20260925 05:01:39.193279 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance
I20260925 05:01:39.193905 31708 external_mini_cluster.cc:1428] Setting key 1de23680c23bdefbb4850b724e67edf3
Sep 25 05:01:39 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312499, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:01:39.202391 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:39.185676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:34065 (local address 127.30.247.62:37869)
0925 05:01:39.186115 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:39.186120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:39.186995 (+ 875us) server_negotiation.cc:407] Connection header received
0925 05:01:39.188828 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:39.188833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:39.188922 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:39.189057 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:39.190512 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:39.191241 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:39.192803 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:39.193249 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:39.193377 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:39.196892 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:39.196898 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:39.196902 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:39.196942 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:39.199458 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:39.200135 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:39.200141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:39.200144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:39.200270 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:39.200746 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:39.200751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:39.200755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:39.201026 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:39.201183 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:39.201823 (+ 640us) server_negotiation.cc:299] Negotiation successful
0925 05:01:39.202035 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":223,"threads_started":1}
I20260925 05:01:39.203297 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:39.186160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.1:34065)
0925 05:01:39.186830 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:39.186866 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:39.188505 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:39.189285 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:39.189293 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:39.189767 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:39.190346 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:39.190363 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:39.191385 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:39.191390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:39.192660 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:39.192677 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:39.193469 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:39.193478 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:39.193713 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:39.194407 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:39.194429 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:39.196685 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:39.199595 (+ 2910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:39.199603 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:39.199618 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:39.200009 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:39.200379 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:39.200385 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:39.200388 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:39.200626 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:39.201158 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:39.201165 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:39.201472 (+ 307us) client_negotiation.cc:769] Sending connection context
0925 05:01:39.201728 (+ 256us) client_negotiation.cc:240] Negotiation successful
0925 05:01:39.202222 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":258,"threads_started":1}
I20260925 05:01:39.204774 7730 heartbeater.cc:347] Connected to a master server at 127.30.247.62:37869
I20260925 05:01:39.205076 7730 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:39.205861 7730 heartbeater.cc:511] Master 127.30.247.62:37869 requested a full tablet report, sending...
I20260925 05:01:39.207868 7541 ts_manager.cc:194] Registered new tserver with Master: bedce651edc54e248c584b27e01e8dd8 (127.30.247.1:38189)
I20260925 05:01:39.209378 7541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.1@KRBTEST.COM'} at 127.30.247.1:34065
WARNING: no policy specified for kudu/127.30.247.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
I20260925 05:01:39.240356 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:0
--local_ip_for_outbound_sockets=127.30.247.2
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:37869
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:39.544481 7736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:39.544742 7736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:39.544893 7736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:39.551699 7736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:39.551815 7736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:39.552017 7736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:01:39.561175 7736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:37869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:39.562278 7736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:39.563477 7736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:39.576241 7742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:39.576296 7741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:39.576213 7744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:39.577342 7736 server_base.cc:1034] running on GCE node
I20260925 05:01:39.577858 7736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:39.578656 7736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:39.579860 7736 hybrid_clock.cc:648] HybridClock initialized: now 1790312499579801 us; error 49 us; skew 500 ppm
I20260925 05:01:39.580179 7736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:39 dist-test-slave-5n86 krb5kdc[7496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312499, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:39.583125 7736 init.cc:377] Logged in from keytab as kudu/127.30.247.2@KRBTEST.COM (short username kudu)
I20260925 05:01:39.584369 7736 webserver.cc:460] Webserver started at http://127.30.247.2:38707/ using document root <none> and password file <none>
I20260925 05:01:39.585029 7736 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:39.585129 7736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:39.585357 7736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:39.587983 7736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/data/instance:
uuid: "c9102522b6344e10ad2556bd7a42f659"
format_stamp: "Formatted at 2026-09-25 05:01:39 on dist-test-slave-5n86"
server_key: "6a562ec548d84879aca93d7ca7efeb5a"
server_key_iv: "de42c3daa7fd2d5c19015e38fe0fdd6b"
server_key_version: "encryptionkey@0"
I20260925 05:01:39.588733 7736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance:
uuid: "c9102522b6344e10ad2556bd7a42f659"
format_stamp: "Formatted at 2026-09-25 05:01:39 on dist-test-slave-5n86"
server_key: "6a562ec548d84879aca93d7ca7efeb5a"
server_key_iv: "de42c3daa7fd2d5c19015e38fe0fdd6b"
server_key_version: "encryptionkey@0"
I20260925 05:01:39.593382 7736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 05:01:39.596417 7751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:39.597667 7736 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 05:01:39.597837 7736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "c9102522b6344e10ad2556bd7a42f659"
format_stamp: "Formatted at 2026-09-25 05:01:39 on dist-test-slave-5n86"
server_key: "6a562ec548d84879aca93d7ca7efeb5a"
server_key_iv: "de42c3daa7fd2d5c19015e38fe0fdd6b"
server_key_version: "encryptionkey@0"
I20260925 05:01:39.598021 7736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:39.610195 7736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:39.614792 7736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:39.615062 7736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:39.616017 7736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:39.617482 7736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:39.617581 7736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:39.617688 7736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:39.617743 7736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:39.653206 7736 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:36601
I20260925 05:01:39.653358 7863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:36601 every 8 connection(s)
I20260925 05:01:39.654462 7736 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:01:39.657881 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 7736
I20260925 05:01:39.658061 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance
I20260925 05:01:39.658640 31708 external_mini_cluster.cc:1428] Setting key 407c04ef62f26253868317568dc5c170
Sep 25 05:01:39 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312499, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:01:39.676626 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:39.656934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:50169 (local address 127.30.247.62:37869)
0925 05:01:39.657142 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:39.657148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:39.658459 (+ 1311us) server_negotiation.cc:407] Connection header received
0925 05:01:39.660653 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:39.660657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:39.660717 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:39.660802 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:39.662819 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:39.663519 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:39.665917 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:39.666307 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:39.666400 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:39.670020 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:39.670029 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:39.670037 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:39.670078 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:39.673571 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:39.674238 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:39.674246 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:39.674253 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:39.674405 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:39.674804 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:39.674811 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:39.674818 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:39.675159 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:39.675307 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:39.675898 (+ 591us) server_negotiation.cc:299] Negotiation successful
0925 05:01:39.676143 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 05:01:39.677232 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:39.657423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.2:50169)
0925 05:01:39.658284 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:39.658334 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:39.660333 (+ 1999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:39.661073 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:39.661085 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:39.661831 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:39.662651 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:39.662679 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:39.663669 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:39.663677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:39.665782 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:39.665800 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:39.666521 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:39.666530 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:39.666845 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:39.667565 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:39.667597 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:39.669820 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:39.673698 (+ 3878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:39.673704 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:39.673714 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:39.674106 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:39.674508 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:39.674511 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:39.674513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:39.674679 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:39.675269 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:39.675274 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:39.675538 (+ 264us) client_negotiation.cc:769] Sending connection context
0925 05:01:39.675745 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 05:01:39.676077 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":302,"threads_started":1}
I20260925 05:01:39.678609 7864 heartbeater.cc:347] Connected to a master server at 127.30.247.62:37869
I20260925 05:01:39.678898 7864 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:39.679562 7864 heartbeater.cc:511] Master 127.30.247.62:37869 requested a full tablet report, sending...
I20260925 05:01:39.681043 7541 ts_manager.cc:194] Registered new tserver with Master: c9102522b6344e10ad2556bd7a42f659 (127.30.247.2:36601)
WARNING: no policy specified for kudu/127.30.247.3@KRBTEST.COM; defaulting to no policy
I20260925 05:01:39.681995 7541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.2@KRBTEST.COM'} at 127.30.247.2:50169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
I20260925 05:01:39.707526 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:0
--local_ip_for_outbound_sockets=127.30.247.3
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:37869
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:40.019106 7869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:40.019362 7869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:40.019450 7869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:40.026507 7869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:40.026600 7869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:40.026707 7869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:01:40.035802 7869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:37869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:40.036975 7869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:40.038177 7869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:40.050606 7875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:40.050690 7874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:40.051119 7877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:40.051438 7869 server_base.cc:1034] running on GCE node
I20260925 05:01:40.051987 7869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:40.052911 7869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:40.054193 7869 hybrid_clock.cc:648] HybridClock initialized: now 1790312500054117 us; error 57 us; skew 500 ppm
I20260925 05:01:40.054513 7869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:40 dist-test-slave-5n86 krb5kdc[7496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312500, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:40.057451 7869 init.cc:377] Logged in from keytab as kudu/127.30.247.3@KRBTEST.COM (short username kudu)
I20260925 05:01:40.058764 7869 webserver.cc:460] Webserver started at http://127.30.247.3:33057/ using document root <none> and password file <none>
I20260925 05:01:40.059405 7869 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:40.059546 7869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:40.059835 7869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:40.062511 7869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/data/instance:
uuid: "4779f334f97b45dc86e37fb3ae58ebe2"
format_stamp: "Formatted at 2026-09-25 05:01:40 on dist-test-slave-5n86"
server_key: "5ca21c654651b730a64e1f2d926e999d"
server_key_iv: "011824a14de61f94062ca4d53826fc7e"
server_key_version: "encryptionkey@0"
I20260925 05:01:40.063262 7869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance:
uuid: "4779f334f97b45dc86e37fb3ae58ebe2"
format_stamp: "Formatted at 2026-09-25 05:01:40 on dist-test-slave-5n86"
server_key: "5ca21c654651b730a64e1f2d926e999d"
server_key_iv: "011824a14de61f94062ca4d53826fc7e"
server_key_version: "encryptionkey@0"
I20260925 05:01:40.068156 7869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 05:01:40.071560 7884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:40.072862 7869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:01:40.073056 7869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "4779f334f97b45dc86e37fb3ae58ebe2"
format_stamp: "Formatted at 2026-09-25 05:01:40 on dist-test-slave-5n86"
server_key: "5ca21c654651b730a64e1f2d926e999d"
server_key_iv: "011824a14de61f94062ca4d53826fc7e"
server_key_version: "encryptionkey@0"
I20260925 05:01:40.073242 7869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:40.095085 7869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:40.099380 7869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:40.099673 7869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:40.100561 7869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:40.101927 7869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:40.102032 7869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:40.102172 7869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:40.102239 7869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:40.132807 7869 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:37565
I20260925 05:01:40.132853 7996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:37565 every 8 connection(s)
I20260925 05:01:40.134047 7869 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:01:40.144212 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 7869
I20260925 05:01:40.144392 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance
I20260925 05:01:40.145028 31708 external_mini_cluster.cc:1428] Setting key 7688364f6c7b9d1a8c643507b844b3b7
Sep 25 05:01:40 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312500, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:01:40.154439 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:40.136608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:34663 (local address 127.30.247.62:37869)
0925 05:01:40.136781 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:40.136785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:40.137857 (+ 1072us) server_negotiation.cc:407] Connection header received
0925 05:01:40.139654 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:40.139660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:40.139753 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:40.139871 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:40.141265 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:40.141982 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:40.143740 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:40.144571 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:40.144715 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:40.148295 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:40.148302 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:40.148307 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:40.148351 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:40.151145 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:40.151856 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:40.151864 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:40.151868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:40.152001 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:40.152523 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:40.152558 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:40.152562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:40.152901 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:40.153067 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:40.153844 (+ 777us) server_negotiation.cc:299] Negotiation successful
0925 05:01:40.154076 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 05:01:40.155340 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:40.137056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.3:34663)
0925 05:01:40.137699 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:40.137739 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:40.139382 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:40.140097 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:40.140105 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:40.140577 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:40.141096 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:40.141117 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:40.142131 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:40.142136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:40.143540 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:40.143556 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:40.144767 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:40.144772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:40.144956 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:40.145653 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:40.145674 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:40.148076 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:40.151286 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:40.151294 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:40.151309 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:40.151729 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:40.152120 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:01:40.152127 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:01:40.152130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:01:40.152381 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:01:40.153038 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:40.153046 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:40.153393 (+ 347us) client_negotiation.cc:769] Sending connection context
0925 05:01:40.153731 (+ 338us) client_negotiation.cc:240] Negotiation successful
0925 05:01:40.154257 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":244,"threads_started":1}
I20260925 05:01:40.156684 7997 heartbeater.cc:347] Connected to a master server at 127.30.247.62:37869
I20260925 05:01:40.156962 7997 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:40.157595 7997 heartbeater.cc:511] Master 127.30.247.62:37869 requested a full tablet report, sending...
I20260925 05:01:40.159197 7540 ts_manager.cc:194] Registered new tserver with Master: 4779f334f97b45dc86e37fb3ae58ebe2 (127.30.247.3:37565)
I20260925 05:01:40.160260 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:01:40.160450 7540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.3@KRBTEST.COM'} at 127.30.247.3:34663
WARNING: no policy specified for oryx/127.30.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.1.keytab.
Entry for principal oryx/127.30.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.1.keytab.
I20260925 05:01:40.212831 7730 heartbeater.cc:503] Master 127.30.247.62:37869 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.30.247.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.2.keytab.
Entry for principal oryx/127.30.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.2.keytab.
WARNING: no policy specified for oryx/127.30.247.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.3.keytab.
Entry for principal oryx/127.30.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.3.keytab.
I20260925 05:01:40.296375 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 7602
Sep 25 05:01:40 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.1@KRBTEST.COM
I20260925 05:01:40.311749 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:40.297515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41896 (local address 127.30.247.1:38189)
0925 05:01:40.298165 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:40.298172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:40.298200 (+ 28us) server_negotiation.cc:407] Connection header received
0925 05:01:40.298262 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:40.298268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:40.298476 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:40.298618 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:40.299957 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:40.301074 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:40.302345 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:40.302899 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:40.302997 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:40.306838 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:40.306848 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:40.306861 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:40.306893 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:40.309427 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:40.310037 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:40.310044 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:40.310046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:40.310178 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:40.310495 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:40.310498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:40.310500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:40.310810 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:40.311030 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:40.311268 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 05:01:40.311450 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":357,"threads_started":1}
I20260925 05:01:40.313058 7725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:40.394915 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 7602
I20260925 05:01:40.411164 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 7736
Sep 25 05:01:40 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.2@KRBTEST.COM
I20260925 05:01:40.426164 8009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:40.412013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.30.247.2:36601)
0925 05:01:40.412522 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:40.412528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:40.412605 (+ 77us) server_negotiation.cc:407] Connection header received
0925 05:01:40.412664 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:40.412670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:40.412878 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:40.413053 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:40.414207 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:40.415198 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:40.416430 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:40.417141 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:40.417279 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:40.421226 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:40.421237 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:40.421250 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:40.421281 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:40.423834 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:40.424407 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:40.424414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:40.424416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:40.424523 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:40.424897 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:40.424900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:40.424902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:40.425200 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:40.425410 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:40.425684 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 05:01:40.425861 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":260,"threads_started":1}
I20260925 05:01:40.427671 7859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:40.507453 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 7736
I20260925 05:01:40.522779 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 7869
Sep 25 05:01:40 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.3@KRBTEST.COM
I20260925 05:01:40.538479 8011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:40.523781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40938 (local address 127.30.247.3:37565)
0925 05:01:40.524413 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:40.524420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:40.524448 (+ 28us) server_negotiation.cc:407] Connection header received
0925 05:01:40.524503 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:40.524509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:40.524779 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:40.524902 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:40.526134 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:40.527198 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:40.528376 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:40.529004 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:40.529113 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:40.533390 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:40.533400 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:40.533413 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:40.533447 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:40.536183 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:40.536868 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:40.536874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:40.536876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:40.536996 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:40.537295 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:40.537298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:40.537300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:40.537595 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:40.537824 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:40.538016 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 05:01:40.538210 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":334,"threads_started":1}
I20260925 05:01:40.539697 7992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:40.620213 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 7869
I20260925 05:01:40.635632 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 7509
I20260925 05:01:40.636806 7571 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:40.735598 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 7509
I20260925 05:01:40.755465 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:37869
--webserver_interface=127.30.247.62
--webserver_port=41137
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:37869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:01:41.062561 8014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:41.062861 8014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:41.063028 8014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:41.069783 8014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:01:41.069916 8014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:41.070021 8014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:41.070129 8014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:01:41.070216 8014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:01:41.079988 8014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:37869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:37869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=41137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:41.081233 8014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:41.082459 8014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:41.093732 8019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:41.093657 8020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:41.093706 8022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:41.094728 8014 server_base.cc:1034] running on GCE node
I20260925 05:01:41.095417 8014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:41.096724 8014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:41.097960 8014 hybrid_clock.cc:648] HybridClock initialized: now 1790312501097918 us; error 46 us; skew 500 ppm
I20260925 05:01:41.098313 8014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:41 dist-test-slave-5n86 krb5kdc[7496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312501, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:41.101297 8014 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:01:41.102632 8014 webserver.cc:460] Webserver started at http://127.30.247.62:41137/ using document root <none> and password file <none>
I20260925 05:01:41.103279 8014 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:41.103423 8014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:41.108381 8014 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:01:41.110978 8029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:41.112248 8014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:01:41.112433 8014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "12a5806c85304822895da797c83d4a11"
format_stamp: "Formatted at 2026-09-25 05:01:38 on dist-test-slave-5n86"
server_key: "6f6388068cdf2a8923b6516c7f3f0fd9"
server_key_iv: "6f09723b8a2a97ab6bb5eb645b57c70d"
server_key_version: "encryptionkey@0"
I20260925 05:01:41.113396 8014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:41.138269 8014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:41.141813 8014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:41.142064 8014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:41.159611 8014 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:37869
I20260925 05:01:41.159644 8080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:37869 every 8 connection(s)
I20260925 05:01:41.160951 8014 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:01:41.164067 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 8014
I20260925 05:01:41.165043 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:38189
--local_ip_for_outbound_sockets=127.30.247.1
--tserver_master_addrs=127.30.247.62:37869
--webserver_port=43295
--webserver_interface=127.30.247.1
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.30.247.1
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:01:41.168998 8081 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:01:41.173662 8081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11: Bootstrap starting.
I20260925 05:01:41.191635 8081 log.cc:824] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11: Log is configured to *not* fsync() on all Append() calls
I20260925 05:01:41.200403 8081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:01:41.200929 8081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11: Bootstrap complete.
I20260925 05:01:41.209718 8081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a5806c85304822895da797c83d4a11" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37869 } }
I20260925 05:01:41.210520 8081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a5806c85304822895da797c83d4a11, State: Initialized, Role: FOLLOWER
I20260925 05:01:41.211216 8081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a5806c85304822895da797c83d4a11" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37869 } }
I20260925 05:01:41.211449 8081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:01:41.211556 8081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:01:41.211752 8081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [term 1 FOLLOWER]: Advancing to term 2
I20260925 05:01:41.215418 8081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a5806c85304822895da797c83d4a11" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37869 } }
I20260925 05:01:41.215900 8081 leader_election.cc:302] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12a5806c85304822895da797c83d4a11; no voters:
I20260925 05:01:41.216604 8081 leader_election.cc:288] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 05:01:41.216734 8085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 05:01:41.219046 8081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:01:41.221094 8085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [term 2 LEADER]: Becoming Leader. State: Replica: 12a5806c85304822895da797c83d4a11, State: Running, Role: LEADER
I20260925 05:01:41.221802 8085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a5806c85304822895da797c83d4a11" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37869 } }
I20260925 05:01:41.224483 8087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12a5806c85304822895da797c83d4a11. Latest consensus state: current_term: 2 leader_uuid: "12a5806c85304822895da797c83d4a11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a5806c85304822895da797c83d4a11" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37869 } } }
I20260925 05:01:41.224857 8087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:41.225353 8086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "12a5806c85304822895da797c83d4a11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a5806c85304822895da797c83d4a11" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37869 } } }
I20260925 05:01:41.225656 8086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11 [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:41.237257 8095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:01:41.244134 8095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:01:41.245527 8095 catalog_manager.cc:1259] Loaded cluster ID: 8ee05980f31b45c38ae9da6de062ce65
I20260925 05:01:41.245673 8095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:01:41.249854 8095 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:01:41.250792 8102 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 05:01:41.251405 8095 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 12a5806c85304822895da797c83d4a11: Loaded TSK: 0
I20260925 05:01:41.252817 8095 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 05:01:41.491973 8083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:41.492221 8083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:41.492369 8083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:41.499437 8083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:41.499534 8083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:41.499696 8083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:01:41.509212 8083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.1.keytab
--principal=oryx/127.30.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:38189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=43295
--tserver_master_addrs=127.30.247.62:37869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:41.510426 8083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:41.511682 8083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:41.526062 8107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:41.526085 8108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:41.526966 8110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:41.527725 8083 server_base.cc:1034] running on GCE node
I20260925 05:01:41.528355 8083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:41.529407 8083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:41.530630 8083 hybrid_clock.cc:648] HybridClock initialized: now 1790312501530578 us; error 43 us; skew 500 ppm
I20260925 05:01:41.530946 8083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:41 dist-test-slave-5n86 krb5kdc[7496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312501, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:41.534159 8083 init.cc:377] Logged in from keytab as oryx/127.30.247.1@KRBTEST.COM (short username oryx)
I20260925 05:01:41.535568 8083 webserver.cc:460] Webserver started at http://127.30.247.1:43295/ using document root <none> and password file <none>
I20260925 05:01:41.536223 8083 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:41.536355 8083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:41.541571 8083 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 05:01:41.544229 8117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:41.545578 8083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:01:41.545753 8083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "bedce651edc54e248c584b27e01e8dd8"
format_stamp: "Formatted at 2026-09-25 05:01:39 on dist-test-slave-5n86"
server_key: "37c81caae811f4d19eaf2158644dc7d9"
server_key_iv: "31bdc6b3d3d75b459993aad259543ff1"
server_key_version: "encryptionkey@0"
I20260925 05:01:41.546662 8083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:41.564370 8083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:41.568182 8083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:41.568441 8083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:41.569397 8083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:41.570744 8083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:41.570863 8083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:41.570996 8083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:41.571081 8083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:41.602509 8083 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:38189
I20260925 05:01:41.602586 8229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:38189 every 8 connection(s)
I20260925 05:01:41.603828 8083 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:01:41.606570 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 8083
I20260925 05:01:41.607340 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:36601
--local_ip_for_outbound_sockets=127.30.247.2
--tserver_master_addrs=127.30.247.62:37869
--webserver_port=38707
--webserver_interface=127.30.247.2
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.30.247.2
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:01:41 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:41 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 05:01:41.951550 8234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:41.951788 8234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:41.951927 8234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:41.960469 8234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:41.960626 8234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:41.960844 8234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:01:41.973191 8234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.2.keytab
--principal=oryx/127.30.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:36601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=38707
--tserver_master_addrs=127.30.247.62:37869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:41.974342 8234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:41.975531 8234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:41.990917 8240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:41.990912 8241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:41.991859 8243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:41.992189 8234 server_base.cc:1034] running on GCE node
I20260925 05:01:41.993188 8234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:41.994658 8234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:41.995939 8234 hybrid_clock.cc:648] HybridClock initialized: now 1790312501995861 us; error 45 us; skew 500 ppm
I20260925 05:01:41.996434 8234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:41 dist-test-slave-5n86 krb5kdc[7496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312501, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:42.001657 8234 init.cc:377] Logged in from keytab as oryx/127.30.247.2@KRBTEST.COM (short username oryx)
I20260925 05:01:42.003844 8234 webserver.cc:460] Webserver started at http://127.30.247.2:38707/ using document root <none> and password file <none>
I20260925 05:01:42.005023 8234 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:42.005231 8234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:42.014464 8234 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.004s
I20260925 05:01:42.018977 8250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:42.021118 8234 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 05:01:42.021347 8234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "c9102522b6344e10ad2556bd7a42f659"
format_stamp: "Formatted at 2026-09-25 05:01:39 on dist-test-slave-5n86"
server_key: "6a562ec548d84879aca93d7ca7efeb5a"
server_key_iv: "de42c3daa7fd2d5c19015e38fe0fdd6b"
server_key_version: "encryptionkey@0"
I20260925 05:01:42.023249 8234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:42.049937 8234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:42.057049 8234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:42.057441 8234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:42.058949 8234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:42.061225 8234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:42.061386 8234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:42.061543 8234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:42.061650 8234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:42.116617 8234 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:36601
I20260925 05:01:42.116729 8362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:36601 every 8 connection(s)
I20260925 05:01:42.118165 8234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:01:42.121172 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 8234
I20260925 05:01:42.122068 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:37565
--local_ip_for_outbound_sockets=127.30.247.3
--tserver_master_addrs=127.30.247.62:37869
--webserver_port=33057
--webserver_interface=127.30.247.3
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.30.247.3
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:01:42 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:42 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 05:01:42.734320 8232 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.116s user 0.004s sys 0.008s
I20260925 05:01:42.736212 8232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:41.606573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.1:43465)
0925 05:01:41.607411 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:41.607706 (+ 295us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:41.610357 (+ 2651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:41.611634 (+ 1277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:41.611647 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:41.612376 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:41.613164 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:41.613191 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:41.614699 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:41.614708 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:41.616304 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:41.616315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:41.617890 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:41.617895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:41.618084 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:41.618744 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:41.618766 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:42.735120 (+1116354us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":336,"threads_started":1}
W20260925 05:01:42.736251 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:41.606553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:43465 (local address 127.30.247.62:37869)
0925 05:01:41.609080 (+ 2527us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:41.609090 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:41.609129 (+ 39us) server_negotiation.cc:407] Connection header received
0925 05:01:41.610846 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:41.610859 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:41.611161 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:41.611423 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:41.613360 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:41.614545 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:41.616433 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:41.617656 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:41.617864 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:42.735496 (+1117632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.1:43465: BlockingRecv error: recv got EOF from 127.30.247.1:43465 (error 108)
Metrics: {"server-negotiator.queue_time_us":2300,"thread_start_us":2088,"threads_started":1}
W20260925 05:01:42.737336 8230 heartbeater.cc:644] Failed to heartbeat to 127.30.247.62:37869 (0 consecutive failures): Not authorized: Failed to ping master at 127.30.247.62:37869: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Sep 25 05:01:42 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:42 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:42.746905 8232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:42.738035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.1:45107)
0925 05:01:42.738610 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:42.738633 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:42.738776 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:42.739252 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:42.739257 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:42.739621 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:42.739776 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:42.739785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:42.740814 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:42.740821 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:42.741743 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:42.741754 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:42.742446 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:42.742452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:42.742622 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:42.743220 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:42.743237 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:42.746601 (+ 3364us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":1155,"threads_started":1}
W20260925 05:01:42.746989 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:42.738438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:45107 (local address 127.30.247.62:37869)
0925 05:01:42.738586 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:42.738594 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:42.738692 (+ 98us) server_negotiation.cc:407] Connection header received
0925 05:01:42.738900 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:42.738907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:42.738995 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:42.739108 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:42.739923 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:42.740685 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:42.741890 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:42.742281 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:42.742417 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:42.746774 (+ 4357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.1:45107: BlockingRecv error: recv got EOF from 127.30.247.1:45107 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 05:01:42 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:42 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:42.756122 8232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:42.747784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.1:36519)
0925 05:01:42.747976 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:42.747994 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:42.748100 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:42.748395 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:42.748399 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:42.748764 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:42.748898 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:42.748905 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:42.750236 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:42.750242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:42.751404 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:42.751415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:42.752074 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:42.752080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:42.752240 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:42.752842 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:42.752869 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:42.755866 (+ 2997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110}
W20260925 05:01:42.756263 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:42.747838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:36519 (local address 127.30.247.62:37869)
0925 05:01:42.747974 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:42.747979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:42.748068 (+ 89us) server_negotiation.cc:407] Connection header received
0925 05:01:42.748203 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:42.748208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:42.748279 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:42.748374 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:42.749136 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:42.750099 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:42.751533 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:42.751933 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:42.752070 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:42.756047 (+ 3977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.1:36519: BlockingRecv error: recv got EOF from 127.30.247.1:36519 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 05:01:42.756628 8230 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 05:01:42.769676 8367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:42.770179 8367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:42.770411 8367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:42.788897 8367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:42.789101 8367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:42.789412 8367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:01:42.813874 8367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/krb5kdc/oryx_127.30.247.3.keytab
--principal=oryx/127.30.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:37565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=33057
--tserver_master_addrs=127.30.247.62:37869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:42.816339 8367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:42.818756 8367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:42.837319 8374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:42.837390 8375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:42.837754 8377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:42.838271 8367 server_base.cc:1034] running on GCE node
I20260925 05:01:42.839247 8367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:42.840759 8367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:42.842048 8367 hybrid_clock.cc:648] HybridClock initialized: now 1790312502841961 us; error 54 us; skew 500 ppm
I20260925 05:01:42.842553 8367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:01:42 dist-test-slave-5n86 krb5kdc[7496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312502, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:01:42.847317 8367 init.cc:377] Logged in from keytab as oryx/127.30.247.3@KRBTEST.COM (short username oryx)
I20260925 05:01:42.848882 8367 webserver.cc:460] Webserver started at http://127.30.247.3:33057/ using document root <none> and password file <none>
I20260925 05:01:42.849539 8367 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:42.849713 8367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:42.855562 8367 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 05:01:42.858762 8384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:42.860379 8367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:01:42.860595 8367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "4779f334f97b45dc86e37fb3ae58ebe2"
format_stamp: "Formatted at 2026-09-25 05:01:40 on dist-test-slave-5n86"
server_key: "5ca21c654651b730a64e1f2d926e999d"
server_key_iv: "011824a14de61f94062ca4d53826fc7e"
server_key_version: "encryptionkey@0"
I20260925 05:01:42.861557 8367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:42.888793 8367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:42.893275 8367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:42.893551 8367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:42.894564 8367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:42.895998 8367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:42.896130 8367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:42.896251 8367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:42.896339 8367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:42.945276 8367 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:37565
I20260925 05:01:42.945333 8496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:37565 every 8 connection(s)
I20260925 05:01:42.946619 8367 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:01:42.956913 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 8367
Sep 25 05:01:42 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:42 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:42.983383 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:42.957837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43444 (local address 127.30.247.62:37869)
0925 05:01:42.970935 (+ 13098us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:42.970939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:42.970952 (+ 13us) server_negotiation.cc:407] Connection header received
0925 05:01:42.970988 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:42.970991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:42.971041 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:42.971152 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:42.972599 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:42.973365 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:42.974573 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:42.974957 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:42.975083 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:42.977594 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:42.977603 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:42.977619 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:42.977650 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:42.980401 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:42.981072 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:42.981087 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:42.981094 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:42.981243 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:01:42.981607 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:01:42.981616 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:01:42.981618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:01:42.982008 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:42.982319 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:42.982741 (+ 422us) server_negotiation.cc:299] Negotiation successful
0925 05:01:42.983096 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13004,"thread_start_us":12861,"threads_started":1}
W20260925 05:01:43.398286 8365 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.250s user 0.007s sys 0.018s
W20260925 05:01:43.399888 8366 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:42.124812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:50503 (local address 127.30.247.62:37869)
0925 05:01:42.138181 (+ 13369us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:42.138191 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:42.138221 (+ 30us) server_negotiation.cc:407] Connection header received
0925 05:01:42.138310 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:42.138319 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:42.138425 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:42.138586 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:42.140694 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:42.142407 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:42.145152 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:42.145999 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:42.146155 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:43.399646 (+1253491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.2:50503: BlockingRecv error: recv got EOF from 127.30.247.2:50503 (error 108)
Metrics: {"server-negotiator.queue_time_us":13192,"thread_start_us":13030,"threads_started":1}
I20260925 05:01:43.401185 8365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:42.122630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.2:50503)
0925 05:01:42.132450 (+ 9820us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:42.132516 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:42.136059 (+ 3543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:42.138878 (+ 2819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:42.138893 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:42.139659 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:42.140428 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:42.140455 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:42.142610 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:42.142618 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:42.144951 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:42.144969 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:42.146316 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:42.146327 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:42.146782 (+ 455us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:42.147813 (+ 1031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:42.147858 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:43.399521 (+1251663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":9370,"thread_start_us":13851,"threads_started":1}
W20260925 05:01:43.402145 8363 heartbeater.cc:644] Failed to heartbeat to 127.30.247.62:37869 (0 consecutive failures): Not authorized: Failed to ping master at 127.30.247.62:37869: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Sep 25 05:01:43 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:43 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:43.414690 8365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:43.402793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.2:44595)
0925 05:01:43.402969 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:43.402989 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:43.403111 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:43.403522 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:43.403531 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:43.404168 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:43.404497 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:43.404513 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:43.405596 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:43.405605 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:43.407871 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:43.407892 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:43.408647 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:43.408657 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:43.408912 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:43.409663 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:43.409698 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:43.414398 (+ 4700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 05:01:43.414824 8366 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:43.402892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:44595 (local address 127.30.247.62:37869)
0925 05:01:43.403071 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:43.403077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:43.403095 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:01:43.403218 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:43.403223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:43.403322 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:43.403422 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:43.404703 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:43.405431 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:43.408038 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:43.408413 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:43.408499 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:43.414564 (+ 6065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.2:44595: BlockingRecv error: recv got EOF from 127.30.247.2:44595 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 05:01:43 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:43 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 05:01:43.424824 8366 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:43.416267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:54153 (local address 127.30.247.62:37869)
0925 05:01:43.416495 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:43.416505 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:43.416528 (+ 23us) server_negotiation.cc:407] Connection header received
0925 05:01:43.416619 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:43.416627 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:43.416734 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:43.416872 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:43.417518 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:43.418239 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:43.419426 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:43.419932 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:43.420053 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:43.424633 (+ 4580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.2:54153: BlockingRecv error: recv got EOF from 127.30.247.2:54153 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 05:01:43.424834 8365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:43.415658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.2:54153)
0925 05:01:43.415835 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:43.415853 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:43.415946 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:43.416911 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:43.416917 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:43.417213 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:43.417350 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:43.417357 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:43.418373 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:43.418379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:43.419290 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:43.419300 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:43.420091 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:43.420096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:43.420251 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:43.420854 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:43.420870 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:43.424398 (+ 3528us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 05:01:43.425278 8363 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 05:01:43 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:43 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:43.773847 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:43.757351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.1:55587)
0925 05:01:43.757875 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:43.757890 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:43.758051 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:43.758406 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:43.758410 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:43.758809 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:43.758950 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:43.758959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:43.760264 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:43.760269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:43.761598 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:43.761610 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:43.762296 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:43.762303 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:43.762469 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:43.769696 (+ 7227us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:43.769714 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:43.773520 (+ 3806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":324,"threads_started":1}
W20260925 05:01:43.773922 8366 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:43.757354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:55587 (local address 127.30.247.62:37869)
0925 05:01:43.757557 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:43.757566 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:43.757974 (+ 408us) server_negotiation.cc:407] Connection header received
0925 05:01:43.758174 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:43.758179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:43.758274 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:43.758374 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:43.759099 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:43.760034 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:43.761730 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:43.762122 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:43.762260 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:43.773705 (+ 11445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.1:55587: BlockingRecv error: recv got EOF from 127.30.247.1:55587 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 05:01:44.089226 8499 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.123s user 0.006s sys 0.018s
W20260925 05:01:44.090225 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:42.949226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:53413 (local address 127.30.247.62:37869)
0925 05:01:42.949465 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:42.949472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:42.950996 (+ 1524us) server_negotiation.cc:407] Connection header received
0925 05:01:42.954830 (+ 3834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:42.954834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:42.954898 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:42.954999 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:42.957860 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:42.959546 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:42.963082 (+ 3536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:42.964086 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:42.964216 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:44.090029 (+1125813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.3:53413: BlockingRecv error: recv got EOF from 127.30.247.3:53413 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
I20260925 05:01:44.090777 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:42.949675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.3:53413)
0925 05:01:42.950759 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:42.950836 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:42.954404 (+ 3568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:42.955383 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:42.955400 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:42.956406 (+ 1006us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:42.957564 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:42.957599 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:42.959857 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:42.959866 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:42.962823 (+ 2957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:42.962845 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:42.964762 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:42.964772 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:42.965231 (+ 459us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:42.966295 (+ 1064us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:42.966339 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:44.089889 (+1123550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":444,"threads_started":1}
W20260925 05:01:44.091606 8497 heartbeater.cc:644] Failed to heartbeat to 127.30.247.62:37869 (0 consecutive failures): Not authorized: Failed to ping master at 127.30.247.62:37869: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Sep 25 05:01:44 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:44 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:44.100227 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:44.092140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.3:43079)
0925 05:01:44.092339 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:44.092366 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:44.092453 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:44.092781 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:44.092784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:44.093101 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:44.093269 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:44.093276 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:44.094250 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:44.094253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:44.095204 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:44.095214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:44.095884 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:44.095888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:44.095990 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:44.096642 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:44.096667 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:44.100039 (+ 3372us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20260925 05:01:44.100579 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:44.092232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:43079 (local address 127.30.247.62:37869)
0925 05:01:44.092440 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:44.092444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:44.092460 (+ 16us) server_negotiation.cc:407] Connection header received
0925 05:01:44.092590 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:44.092595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:44.092666 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:44.092740 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:44.093399 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:44.094143 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:44.095314 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:44.095748 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:44.095839 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:44.100308 (+ 4469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.3:43079: BlockingRecv error: recv got EOF from 127.30.247.3:43079 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Sep 25 05:01:44 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:44 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:44.108338 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:44.100894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.3:43675)
0925 05:01:44.101093 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:44.101113 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:44.101217 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:44.101525 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:44.101529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:44.101840 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:44.101973 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:44.101981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:44.102935 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:44.102941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:44.103864 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:44.103875 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:44.104580 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:44.104586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:44.104772 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:44.105236 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:44.105252 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:44.108170 (+ 2918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 05:01:44.108516 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:44.100952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:43675 (local address 127.30.247.62:37869)
0925 05:01:44.101093 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:44.101100 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:44.101164 (+ 64us) server_negotiation.cc:407] Connection header received
0925 05:01:44.101336 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:44.101341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:44.101409 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:44.101504 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:44.102111 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:44.102801 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:44.104002 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:44.104417 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:44.104569 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:44.108337 (+ 3768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.3:43675: BlockingRecv error: recv got EOF from 127.30.247.3:43675 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 05:01:44.108641 8497 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 05:01:44 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:44 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:44.435127 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:44.426013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.2:38841)
0925 05:01:44.426707 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:44.426724 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:44.426839 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:44.427163 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:44.427167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:44.427579 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:44.427730 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:44.427739 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:44.428809 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:44.428813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:44.429840 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:44.429852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:44.430650 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:44.430654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:44.430771 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:44.431391 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:44.431408 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:44.434803 (+ 3395us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":452,"threads_started":1}
W20260925 05:01:44.435158 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:44.426066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:38841 (local address 127.30.247.62:37869)
0925 05:01:44.426292 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:44.426301 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:44.426809 (+ 508us) server_negotiation.cc:407] Connection header received
0925 05:01:44.426939 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:44.426942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:44.427032 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:44.427113 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:44.427882 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:44.428658 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:44.430081 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:44.430493 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:44.430607 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:44.434970 (+ 4363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.2:38841: BlockingRecv error: recv got EOF from 127.30.247.2:38841 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 05:01:44 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:44 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:44.783826 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:44.774931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.1:57017)
0925 05:01:44.775436 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:44.775452 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:44.775578 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:44.775972 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:44.775978 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:44.776528 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:44.776787 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:44.776797 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:44.778026 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:44.778030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:44.778945 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:44.778955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:44.779662 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:44.779668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:44.779831 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:44.780389 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:44.780407 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:44.783597 (+ 3190us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":292,"threads_started":1}
W20260925 05:01:44.783979 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:44.774987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:57017 (local address 127.30.247.62:37869)
0925 05:01:44.775159 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:44.775164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:44.775524 (+ 360us) server_negotiation.cc:407] Connection header received
0925 05:01:44.775715 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:44.775720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:44.775811 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:44.775923 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:44.776973 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:44.777894 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:44.779077 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:44.779484 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:44.779580 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:44.783738 (+ 4158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.1:57017: BlockingRecv error: recv got EOF from 127.30.247.1:57017 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 05:01:45 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:45 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:45.117594 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:45.109147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.3:54809)
0925 05:01:45.109641 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:45.109657 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:45.109766 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:45.110026 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:45.110029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:45.110396 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:45.110529 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:45.110537 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:45.111475 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:45.111478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:45.112405 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:45.112415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:45.113147 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:45.113152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:45.113289 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:45.113903 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:45.113921 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:45.117326 (+ 3405us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":274,"threads_started":1}
W20260925 05:01:45.117795 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:45.109262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:54809 (local address 127.30.247.62:37869)
0925 05:01:45.109507 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:45.109513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:45.109734 (+ 221us) server_negotiation.cc:407] Connection header received
0925 05:01:45.109861 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:45.109864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:45.109936 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:45.110011 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:45.110644 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:45.111365 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:45.112572 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:45.112968 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:45.113059 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:45.117561 (+ 4502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.3:54809: BlockingRecv error: recv got EOF from 127.30.247.3:54809 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 25 05:01:45 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:45 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:45.445262 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:45.436017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.2:37995)
0925 05:01:45.436607 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:45.436624 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:45.436851 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:45.437202 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:45.437206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:45.437618 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:45.437791 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:45.437799 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:45.438818 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:45.438824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:45.439724 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:45.439734 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:45.440418 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:45.440424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:45.440654 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:45.441278 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:45.441297 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:45.444888 (+ 3591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":312,"threads_started":1}
W20260925 05:01:45.445328 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:45.436118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:37995 (local address 127.30.247.62:37869)
0925 05:01:45.436291 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:45.436295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:45.436739 (+ 444us) server_negotiation.cc:407] Connection header received
0925 05:01:45.436988 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:45.437001 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:45.437065 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:45.437167 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:45.437939 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:45.438672 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:45.439865 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:45.440258 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:45.440383 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:45.445107 (+ 4724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.2:37995: BlockingRecv error: recv got EOF from 127.30.247.2:37995 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 05:01:45 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:45 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:45.793380 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:45.784776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.1:52227)
0925 05:01:45.785507 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:45.785523 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:45.785634 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:45.785916 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:45.785919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:45.786269 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:45.786434 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:45.786441 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:45.787394 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:45.787398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:45.788283 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:45.788294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:45.789045 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:45.789050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:45.789204 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:45.789825 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:45.789843 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:45.793161 (+ 3318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":477,"threads_started":1}
W20260925 05:01:45.793568 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:45.784871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:52227 (local address 127.30.247.62:37869)
0925 05:01:45.785111 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:45.785121 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:45.785606 (+ 485us) server_negotiation.cc:407] Connection header received
0925 05:01:45.785728 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:45.785732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:45.785793 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:45.785871 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:45.786565 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:45.787275 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:45.788445 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:45.788882 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:45.788984 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:45.793346 (+ 4362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.1:52227: BlockingRecv error: recv got EOF from 127.30.247.1:52227 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 05:01:46 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:46 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:46.128041 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:46.118695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.3:50675)
0925 05:01:46.119226 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:46.119243 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:46.119388 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:46.119733 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:46.119737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:46.120201 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:46.120398 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:46.120407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:46.121658 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:46.121663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:46.123060 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:46.123072 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:46.123745 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:46.123752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:46.123884 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:46.124493 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:46.124509 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:46.127716 (+ 3207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":254,"threads_started":1}
W20260925 05:01:46.128114 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:46.118751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:50675 (local address 127.30.247.62:37869)
0925 05:01:46.118977 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:46.118985 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:46.119318 (+ 333us) server_negotiation.cc:407] Connection header received
0925 05:01:46.119503 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:46.119508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:46.119601 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:46.119702 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:46.120553 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:46.121463 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:46.123167 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:46.123581 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:46.123679 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:46.127890 (+ 4211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.3:50675: BlockingRecv error: recv got EOF from 127.30.247.3:50675 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 05:01:46 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:46 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:46.455061 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:46.446327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.2:48911)
0925 05:01:46.446842 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:46.446858 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:46.446967 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:46.447280 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:46.447283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:46.447737 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:46.447959 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:46.447974 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:46.449428 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:46.449432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:46.450329 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:46.450340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:46.451070 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:46.451074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:46.451211 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:46.451748 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:46.451764 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:46.454839 (+ 3075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":276,"threads_started":1}
W20260925 05:01:46.455233 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:46.446402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:48911 (local address 127.30.247.62:37869)
0925 05:01:46.446557 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:46.446562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:46.446926 (+ 364us) server_negotiation.cc:407] Connection header received
0925 05:01:46.447058 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:46.447061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:46.447118 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:46.447251 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:46.448105 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:46.449268 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:46.450474 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:46.450951 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:46.451043 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:46.455020 (+ 3977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.2:48911: BlockingRecv error: recv got EOF from 127.30.247.2:48911 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 05:01:46 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:46 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:46.802767 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:46.794350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.1:35905)
0925 05:01:46.794806 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:46.794822 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:46.794929 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:46.795207 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:46.795210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:46.795559 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:46.795729 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:46.795737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:46.796753 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:46.796756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:46.797731 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:46.797747 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:46.798520 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:46.798529 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:46.798722 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:46.799325 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:46.799342 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:46.802534 (+ 3192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":238,"threads_started":1}
W20260925 05:01:46.802922 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:46.794416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:35905 (local address 127.30.247.62:37869)
0925 05:01:46.794642 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:46.794648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:46.794896 (+ 248us) server_negotiation.cc:407] Connection header received
0925 05:01:46.795009 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:46.795013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:46.795075 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:46.795154 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:46.795849 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:46.796640 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:46.797875 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:46.798356 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:46.798482 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:46.802699 (+ 4217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.1:35905: BlockingRecv error: recv got EOF from 127.30.247.1:35905 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 05:01:47 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:47 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:47.138420 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:47.129345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.3:59073)
0925 05:01:47.130238 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:47.130255 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:47.130363 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:47.130702 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:47.130707 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:47.131160 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:47.131326 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:47.131335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:47.132348 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:47.132353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:47.133289 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:47.133300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:47.134040 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:47.134046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:47.134227 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:47.134853 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:47.134870 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:47.138093 (+ 3223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":609,"threads_started":1}
W20260925 05:01:47.138520 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:47.129419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:59073 (local address 127.30.247.62:37869)
0925 05:01:47.129610 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:47.129617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:47.130334 (+ 717us) server_negotiation.cc:407] Connection header received
0925 05:01:47.130475 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:47.130479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:47.130566 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:47.130670 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:47.131474 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:47.132193 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:47.133430 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:47.133899 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:47.134029 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:47.138280 (+ 4251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.3:59073: BlockingRecv error: recv got EOF from 127.30.247.3:59073 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 05:01:47 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:47 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:47.465056 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:47.456030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.2:33967)
0925 05:01:47.456556 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:47.456571 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:47.456725 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:47.457082 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:47.457088 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:47.457523 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:47.457750 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:47.457766 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:47.458857 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:47.458865 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:47.459831 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:47.459842 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:47.460515 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:47.460521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:47.460748 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:47.461386 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:47.461403 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:47.464672 (+ 3269us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":260,"threads_started":1}
W20260925 05:01:47.465135 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:47.456060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:33967 (local address 127.30.247.62:37869)
0925 05:01:47.456269 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:47.456278 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:47.456657 (+ 379us) server_negotiation.cc:407] Connection header received
0925 05:01:47.456848 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:47.456859 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:47.456919 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:47.457022 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:47.457868 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:47.458733 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:47.459972 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:47.460372 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:47.460504 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:47.464876 (+ 4372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.2:33967: BlockingRecv error: recv got EOF from 127.30.247.2:33967 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 05:01:47 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:47 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:47.813164 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:47.803604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.1:39323)
0925 05:01:47.804257 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:47.804273 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:47.804428 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:47.804812 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:47.804818 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:47.805247 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:47.805389 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:47.805397 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:47.806448 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:47.806454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:47.807480 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:47.807496 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:47.808455 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:47.808464 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:47.808709 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:47.809354 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:47.809371 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:47.812847 (+ 3476us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":293,"threads_started":1}
W20260925 05:01:47.813277 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:47.803849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:39323 (local address 127.30.247.62:37869)
0925 05:01:47.804122 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:47.804128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:47.804360 (+ 232us) server_negotiation.cc:407] Connection header received
0925 05:01:47.804593 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:47.804598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:47.804674 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:47.804776 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:47.805525 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:47.806311 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:47.807644 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:47.808242 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:47.808437 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:47.813045 (+ 4608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.1:39323: BlockingRecv error: recv got EOF from 127.30.247.1:39323 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 25 05:01:48 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:48 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:48.148192 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:48.139389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.3:33227)
0925 05:01:48.139885 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:48.139901 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:48.140009 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:48.140308 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:48.140311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:48.140697 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:48.140854 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:48.140862 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:48.141846 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:48.141849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:48.142933 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:48.142944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:48.143619 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:48.143623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:48.143738 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:48.144370 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:48.144397 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:48.147814 (+ 3417us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":286,"threads_started":1}
W20260925 05:01:48.148325 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:48.139565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:33227 (local address 127.30.247.62:37869)
0925 05:01:48.139793 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:48.139799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:48.139976 (+ 177us) server_negotiation.cc:407] Connection header received
0925 05:01:48.140094 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:48.140097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:48.140174 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:48.140252 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:48.141007 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:48.141723 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:48.143086 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:48.143482 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:48.143593 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:48.148033 (+ 4440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.3:33227: BlockingRecv error: recv got EOF from 127.30.247.3:33227 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 25 05:01:48 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:48 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:48.474834 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:48.466012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.2:35195)
0925 05:01:48.466559 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:48.466575 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:48.466721 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:48.467051 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:48.467057 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:48.467516 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:48.467666 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:48.467675 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:48.468685 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:48.468692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:48.469636 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:48.469648 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:48.470319 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:48.470325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:48.470511 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:48.471099 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:48.471115 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:48.474619 (+ 3504us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":277,"threads_started":1}
W20260925 05:01:48.475044 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:48.466199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:35195 (local address 127.30.247.62:37869)
0925 05:01:48.466421 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:48.466427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:48.466655 (+ 228us) server_negotiation.cc:407] Connection header received
0925 05:01:48.466846 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:48.466851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:48.466917 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:48.467016 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:48.467802 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:48.468499 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:48.469776 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:48.470175 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:48.470298 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:48.474827 (+ 4529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.2:35195: BlockingRecv error: recv got EOF from 127.30.247.2:35195 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 05:01:48 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:48 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:48.829005 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:48.814157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.1:38249)
0925 05:01:48.814813 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:48.814835 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:48.815001 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:48.815372 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:48.815377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:48.815836 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:48.815996 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:48.816004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:48.817068 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:48.817075 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:48.818003 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:48.818015 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:48.818734 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:48.818740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:48.818913 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:48.825499 (+ 6586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:48.825516 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:48.828780 (+ 3264us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":386,"threads_started":1}
W20260925 05:01:48.829231 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:48.814228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:38249 (local address 127.30.247.62:37869)
0925 05:01:48.814472 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:48.814482 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:48.814930 (+ 448us) server_negotiation.cc:407] Connection header received
0925 05:01:48.815147 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:48.815152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:48.815240 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:48.815339 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:48.816135 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:48.816932 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:48.818143 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:48.818580 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:48.818705 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:48.828956 (+ 10251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.1:38249: BlockingRecv error: recv got EOF from 127.30.247.1:38249 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 05:01:49 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:49 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:49.158668 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:49.149274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.3:55209)
0925 05:01:49.149844 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:49.149861 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:49.150004 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:49.150313 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:49.150316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:49.150714 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:49.150857 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:49.150865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:49.151944 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:49.151948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:49.153132 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:49.153146 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:49.154126 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:49.154131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:49.154321 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:49.154992 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:49.155009 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:49.158436 (+ 3427us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":342,"threads_started":1}
W20260925 05:01:49.158921 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:49.149391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:55209 (local address 127.30.247.62:37869)
0925 05:01:49.149561 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:49.149566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:49.149932 (+ 366us) server_negotiation.cc:407] Connection header received
0925 05:01:49.150128 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:49.150132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:49.150192 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:49.150274 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:49.150978 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:49.151808 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:49.153286 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:49.153955 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:49.154075 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:49.158640 (+ 4565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.3:55209: BlockingRecv error: recv got EOF from 127.30.247.3:55209 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 05:01:49 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:49 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:49.484627 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:49.475879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.2:34023)
0925 05:01:49.476333 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:49.476349 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:49.476460 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:49.476809 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:49.476812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:49.477179 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:49.477343 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:49.477351 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:49.478365 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:49.478369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:49.479274 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:49.479285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:49.479953 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:49.479957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:49.480075 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:49.480733 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:49.480751 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:49.484266 (+ 3515us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":263,"threads_started":1}
W20260925 05:01:49.484817 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:49.476025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:34023 (local address 127.30.247.62:37869)
0925 05:01:49.476303 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:49.476310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:49.476425 (+ 115us) server_negotiation.cc:407] Connection header received
0925 05:01:49.476613 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:49.476617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:49.476680 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:49.476761 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:49.477493 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:49.478236 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:49.479410 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:49.479813 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:49.479907 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:49.484539 (+ 4632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.2:34023: BlockingRecv error: recv got EOF from 127.30.247.2:34023 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Sep 25 05:01:49 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:49 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:49.839664 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:49.830020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.1:41099)
0925 05:01:49.830693 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:49.830711 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:49.830887 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:49.831241 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:49.831245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:49.831757 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:49.831935 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:49.831944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:49.833185 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:49.833193 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:49.834446 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:49.834458 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:49.835183 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:49.835190 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:49.835318 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:49.835901 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:49.835920 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:49.839301 (+ 3381us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":257,"threads_started":1}
W20260925 05:01:49.839720 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:49.830111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:41099 (local address 127.30.247.62:37869)
0925 05:01:49.830338 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:49.830348 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:49.830816 (+ 468us) server_negotiation.cc:407] Connection header received
0925 05:01:49.831023 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:49.831029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:49.831105 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:49.831209 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:49.832078 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:49.833011 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:49.834581 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:49.834989 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:49.835083 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:49.839493 (+ 4410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.1:41099: BlockingRecv error: recv got EOF from 127.30.247.1:41099 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 05:01:50 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:50 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:50.169365 8521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:50.159646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.3:38535)
0925 05:01:50.160168 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:50.160184 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:50.160291 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:50.160659 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:50.160664 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:50.161184 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:50.161405 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:50.161415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:50.162819 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:50.162823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:50.163731 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:50.163742 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:50.164503 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:50.164509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:50.164728 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:50.165403 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:50.165420 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:50.169139 (+ 3719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":299,"threads_started":1}
W20260925 05:01:50.169528 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:50.159745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:38535 (local address 127.30.247.62:37869)
0925 05:01:50.159961 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:50.159967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:50.160267 (+ 300us) server_negotiation.cc:407] Connection header received
0925 05:01:50.160413 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:50.160419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:50.160500 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:50.160620 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:50.161540 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:50.162674 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:50.163893 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:50.164355 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:50.164489 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:50.169326 (+ 4837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.3:38535: BlockingRecv error: recv got EOF from 127.30.247.3:38535 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 05:01:50 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:50 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:50.496156 8522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:50.485676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.2:53723)
0925 05:01:50.486311 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:50.486327 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:50.486438 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:50.486837 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:50.486840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:50.487251 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:50.487410 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:50.487419 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:50.489094 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:50.489099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:50.490035 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:50.490054 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:50.491202 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:50.491207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:50.491323 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:50.492301 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:50.492328 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:50.495921 (+ 3593us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":300,"threads_started":1}
W20260925 05:01:50.496596 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:50.485739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:53723 (local address 127.30.247.62:37869)
0925 05:01:50.486033 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:50.486049 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:50.486411 (+ 362us) server_negotiation.cc:407] Connection header received
0925 05:01:50.486581 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:50.486588 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:50.486681 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:50.486788 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:50.487558 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:50.488925 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:50.490186 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:50.491010 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:50.491169 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:50.496258 (+ 5089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.2:53723: BlockingRecv error: recv got EOF from 127.30.247.2:53723 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 25 05:01:50 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:50 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:50.849695 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:50.840527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.1:38375)
0925 05:01:50.841123 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:50.841138 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:50.841260 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:50.841534 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:50.841537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:50.841904 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:50.842065 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:50.842073 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:50.843046 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:50.843050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:50.844110 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:50.844123 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:50.845110 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:50.845116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:50.845295 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:50.845842 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:50.845858 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:50.849334 (+ 3476us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":282,"threads_started":1}
W20260925 05:01:50.849817 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:50.840653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:38375 (local address 127.30.247.62:37869)
0925 05:01:50.840816 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:50.840821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:50.841214 (+ 393us) server_negotiation.cc:407] Connection header received
0925 05:01:50.841355 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:50.841358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:50.841416 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:50.841497 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:50.842194 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:50.842912 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:50.844257 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:50.844934 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:50.845054 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:50.849557 (+ 4503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.1:38375: BlockingRecv error: recv got EOF from 127.30.247.1:38375 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 05:01:51 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:51 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:51.179032 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:51.170338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.3:45893)
0925 05:01:51.170931 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:51.170948 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:51.171065 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:51.171359 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:51.171362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:51.171741 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:51.171902 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:51.171909 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:51.172873 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:51.172877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:51.173921 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:51.173936 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:51.174778 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:51.174786 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:51.174959 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:51.175649 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:51.175665 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:51.178802 (+ 3137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":266,"threads_started":1}
W20260925 05:01:51.179206 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:51.170492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:45893 (local address 127.30.247.62:37869)
0925 05:01:51.170658 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:51.170662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:51.171021 (+ 359us) server_negotiation.cc:407] Connection header received
0925 05:01:51.171179 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:51.171183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:51.171242 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:51.171321 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:51.172015 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:51.172749 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:51.174081 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:51.174621 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:51.174713 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:51.178978 (+ 4265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.3:45893: BlockingRecv error: recv got EOF from 127.30.247.3:45893 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 05:01:51 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:51 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:51.506116 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:51.497258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.2:43863)
0925 05:01:51.497748 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:51.497764 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:51.497893 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:51.498180 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:51.498183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:51.498576 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:51.498735 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:51.498743 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:51.500080 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:51.500088 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:51.501175 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:51.501187 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:51.501911 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:51.501917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:51.502071 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:51.502650 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:51.502666 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:51.505871 (+ 3205us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":261,"threads_started":1}
W20260925 05:01:51.506301 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:51.497331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:43863 (local address 127.30.247.62:37869)
0925 05:01:51.497572 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:51.497578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:51.497844 (+ 266us) server_negotiation.cc:407] Connection header received
0925 05:01:51.497991 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:51.497995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:51.498055 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:51.498135 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:51.498887 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:51.499928 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:51.501318 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:51.501733 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:51.501872 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:51.506051 (+ 4179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.2:43863: BlockingRecv error: recv got EOF from 127.30.247.2:43863 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 05:01:51 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:51 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:51.859556 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:51.850698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.1:35183)
0925 05:01:51.851232 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:51.851248 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:51.851404 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:51.851830 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:51.851835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:51.852350 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:51.852544 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:51.852590 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:51.853584 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:51.853588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:51.854510 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:51.854520 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:51.855126 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:51.855130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:51.855257 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:51.855872 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:51.855889 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:51.859324 (+ 3435us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":281,"threads_started":1}
W20260925 05:01:51.859771 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:51.850757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:35183 (local address 127.30.247.62:37869)
0925 05:01:51.850970 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:51.850980 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:51.851345 (+ 365us) server_negotiation.cc:407] Connection header received
0925 05:01:51.851548 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:51.851552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:51.851671 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:51.851768 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:51.852717 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:51.853473 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:51.854634 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:51.855006 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:51.855098 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:51.859502 (+ 4404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.1:35183: BlockingRecv error: recv got EOF from 127.30.247.1:35183 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 05:01:52 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:52 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:52.188839 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:52.179882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.3:43779)
0925 05:01:52.180400 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:52.180415 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:52.180580 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:52.180904 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:52.180909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:52.181314 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:52.181465 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:52.181474 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:52.182452 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:52.182459 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:52.183381 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:52.183392 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:52.184253 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:52.184258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:52.184427 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:52.185194 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:52.185210 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:52.188440 (+ 3230us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":269,"threads_started":1}
W20260925 05:01:52.188846 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:52.179965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:43779 (local address 127.30.247.62:37869)
0925 05:01:52.180236 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:52.180242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:52.180499 (+ 257us) server_negotiation.cc:407] Connection header received
0925 05:01:52.180717 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:52.180721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:52.180781 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:52.180880 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:52.181603 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:52.182308 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:52.183516 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:52.184048 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:52.184221 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:52.188682 (+ 4461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.3:43779: BlockingRecv error: recv got EOF from 127.30.247.3:43779 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 25 05:01:52 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:52 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:52.516773 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:52.507040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.2:49837)
0925 05:01:52.507569 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:52.507585 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:52.507708 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:52.507994 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:52.507997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:52.508384 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:52.508522 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:52.508529 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:52.509506 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:52.509510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:52.510675 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:52.510696 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:52.511649 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:52.511655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:52.511840 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:52.512473 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:52.512491 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:52.516377 (+ 3886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":301,"threads_started":1}
W20260925 05:01:52.516840 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:52.507176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:49837 (local address 127.30.247.62:37869)
0925 05:01:52.507367 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:52.507371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:52.507701 (+ 330us) server_negotiation.cc:407] Connection header received
0925 05:01:52.507807 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:52.507811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:52.507869 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:52.507952 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:52.508673 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:52.509385 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:52.510899 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:52.511468 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:52.511621 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:52.516584 (+ 4963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.2:49837: BlockingRecv error: recv got EOF from 127.30.247.2:49837 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 05:01:52 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:52 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:52.869989 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:52.860513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.1:60857)
0925 05:01:52.861080 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:52.861095 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:52.861283 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:52.861981 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:52.861986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:52.862477 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:52.862676 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:52.862706 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:52.863699 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:52.863703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:52.864901 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:52.864913 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:52.865583 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:52.865591 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:52.865736 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:52.866395 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:52.866412 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:52.869764 (+ 3352us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":277,"threads_started":1}
W20260925 05:01:52.870220 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:52.860648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:60857 (local address 127.30.247.62:37869)
0925 05:01:52.860816 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:52.860821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:52.861210 (+ 389us) server_negotiation.cc:407] Connection header received
0925 05:01:52.861553 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:52.861557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:52.861627 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:52.861733 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:52.862846 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:52.863556 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:52.865022 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:52.865440 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:52.865529 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:52.869935 (+ 4406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.1:60857: BlockingRecv error: recv got EOF from 127.30.247.1:60857 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 05:01:53 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:53 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:53.198529 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:53.189855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.3:59149)
0925 05:01:53.190446 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:53.190466 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:53.190624 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:53.190970 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:53.190974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:53.191386 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:53.191538 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:53.191546 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:53.192528 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:53.192563 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:53.193506 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:53.193517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:53.194208 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:53.194213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:53.194394 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:53.194925 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:53.194942 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:53.198209 (+ 3267us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":256,"threads_started":1}
W20260925 05:01:53.198660 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:53.189945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:59149 (local address 127.30.247.62:37869)
0925 05:01:53.190161 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:53.190169 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:53.190559 (+ 390us) server_negotiation.cc:407] Connection header received
0925 05:01:53.190761 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:53.190766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:53.190841 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:53.190940 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:53.191685 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:53.192398 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:53.193655 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:53.194057 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:53.194194 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:53.198383 (+ 4189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.3:59149: BlockingRecv error: recv got EOF from 127.30.247.3:59149 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 05:01:53 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:53 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:53.527037 8531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:53.517715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.2:40767)
0925 05:01:53.518243 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:53.518259 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:53.518370 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:53.518648 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:53.518651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:53.518971 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:53.519149 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:53.519206 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:53.520405 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:53.520413 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:53.521625 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:53.521637 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:53.522487 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:53.522491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:53.522611 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:53.523226 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:53.523290 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:53.526808 (+ 3518us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":247,"threads_started":1}
W20260925 05:01:53.527182 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:53.517758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:40767 (local address 127.30.247.62:37869)
0925 05:01:53.517990 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:53.517997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:53.518333 (+ 336us) server_negotiation.cc:407] Connection header received
0925 05:01:53.518466 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:53.518469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:53.518528 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:53.518610 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:53.519322 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:53.520283 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:53.521764 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:53.522329 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:53.522434 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:53.526969 (+ 4535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.2:40767: BlockingRecv error: recv got EOF from 127.30.247.2:40767 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 05:01:53 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:53 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:53.885258 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:53.870917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.1:51799)
0925 05:01:53.871473 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:53.871494 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:53.871636 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:53.872005 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:53.872013 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:53.872646 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:53.872858 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:53.872869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:53.873887 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:53.873895 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:53.874827 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:53.874838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:53.875571 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:53.875576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:53.875759 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:53.881460 (+ 5701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:53.881479 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:53.885045 (+ 3566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":324,"threads_started":1}
W20260925 05:01:53.885468 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:53.871182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:51799 (local address 127.30.247.62:37869)
0925 05:01:53.871444 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:53.871450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:53.871609 (+ 159us) server_negotiation.cc:407] Connection header received
0925 05:01:53.871769 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:53.871775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:53.871854 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:53.871966 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:53.873007 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:53.873756 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:53.874971 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:53.875414 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:53.875543 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:53.885215 (+ 9672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.1:51799: BlockingRecv error: recv got EOF from 127.30.247.1:51799 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Sep 25 05:01:54 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:54 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:54.207683 8533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:54.199455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.3:48451)
0925 05:01:54.199909 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:54.199924 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:54.200030 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:54.200382 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:54.200386 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:54.200900 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:54.201057 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:54.201066 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:54.202077 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:54.202086 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:54.203023 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:54.203033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:54.203699 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:54.203704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:54.203872 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:54.204396 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:54.204412 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:54.207461 (+ 3049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":259,"threads_started":1}
W20260925 05:01:54.207837 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:54.199480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:48451 (local address 127.30.247.62:37869)
0925 05:01:54.199681 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:54.199691 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:54.199999 (+ 308us) server_negotiation.cc:407] Connection header received
0925 05:01:54.200172 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:54.200177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:54.200250 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:54.200355 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:54.201193 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:54.201954 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:54.203167 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:54.203556 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:54.203691 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:54.207619 (+ 3928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.3:48451: BlockingRecv error: recv got EOF from 127.30.247.3:48451 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 05:01:54 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:54 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:54.536953 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:54.528135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.2:59721)
0925 05:01:54.528684 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:54.528703 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:54.528852 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:54.529262 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:54.529269 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:54.529664 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:54.529827 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:54.529835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:54.530913 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:54.530917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:54.531881 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:54.531892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:54.532578 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:54.532587 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:54.532725 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:54.533330 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:54.533346 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:54.536639 (+ 3293us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":265,"threads_started":1}
W20260925 05:01:54.537096 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:54.528198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:59721 (local address 127.30.247.62:37869)
0925 05:01:54.528419 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:54.528427 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:54.528814 (+ 387us) server_negotiation.cc:407] Connection header received
0925 05:01:54.528979 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:54.528987 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:54.529060 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:54.529197 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:54.529950 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:54.530788 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:54.531993 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:54.532409 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:54.532497 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:54.536837 (+ 4340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.2:59721: BlockingRecv error: recv got EOF from 127.30.247.2:59721 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 05:01:54 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:54 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:54.895186 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:54.886147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.1:57341)
0925 05:01:54.886674 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:54.886691 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:54.886815 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:54.887096 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:54.887099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:54.887510 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:54.887697 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:54.887705 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:54.888669 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:54.888673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:54.889610 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:54.889621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:54.890521 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:54.890528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:54.890705 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:54.891457 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:54.891479 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:54.894849 (+ 3370us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":326,"threads_started":1}
W20260925 05:01:54.895274 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:54.886335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:57341 (local address 127.30.247.62:37869)
0925 05:01:54.886633 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:54.886641 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:54.886765 (+ 124us) server_negotiation.cc:407] Connection header received
0925 05:01:54.886913 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:54.886917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:54.886975 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:54.887051 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:54.887818 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:54.888484 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:54.889734 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:54.890343 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:54.890476 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:54.895040 (+ 4564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.1:57341: BlockingRecv error: recv got EOF from 127.30.247.1:57341 (error 108)
Metrics: {"server-negotiator.queue_time_us":172}
Sep 25 05:01:55 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:55 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:55.217020 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:55.208685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.3:36507)
0925 05:01:55.209160 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:55.209176 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:55.209290 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:55.209618 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:55.209622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:55.210009 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:55.210200 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:55.210208 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:55.211162 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:55.211166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:55.212061 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:55.212072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:55.212838 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:55.212842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:55.212975 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:55.213525 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:55.213542 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:55.216729 (+ 3187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":256,"threads_started":1}
W20260925 05:01:55.217195 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:55.208722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:36507 (local address 127.30.247.62:37869)
0925 05:01:55.208936 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:55.208945 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:55.209278 (+ 333us) server_negotiation.cc:407] Connection header received
0925 05:01:55.209387 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:55.209391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:55.209453 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:55.209528 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:55.210337 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:55.211034 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:55.212203 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:55.212700 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:55.212794 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:55.216952 (+ 4158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.3:36507: BlockingRecv error: recv got EOF from 127.30.247.3:36507 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 05:01:55 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:55 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:55.546442 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:55.537860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.2:38555)
0925 05:01:55.538382 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:55.538398 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:55.538504 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:55.538818 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:55.538821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:55.539241 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:55.539400 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:55.539408 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:55.540397 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:55.540401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:55.541312 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:55.541323 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:55.541988 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:55.541993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:55.542138 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:55.542769 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:55.542785 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:55.546174 (+ 3389us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":297,"threads_started":1}
W20260925 05:01:55.546730 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:55.537912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:38555 (local address 127.30.247.62:37869)
0925 05:01:55.538128 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:55.538133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:55.538469 (+ 336us) server_negotiation.cc:407] Connection header received
0925 05:01:55.538604 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:55.538607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:55.538683 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:55.538765 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:55.539539 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:55.540266 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:55.541430 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:55.541840 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:55.541930 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:55.546453 (+ 4523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.2:38555: BlockingRecv error: recv got EOF from 127.30.247.2:38555 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 25 05:01:55 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:55 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:55.904901 8538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:55.896066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.1:44003)
0925 05:01:55.896570 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:55.896585 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:55.896694 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:55.896989 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:55.896992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:55.897402 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:55.897539 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:55.897546 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:55.898583 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:55.898587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:55.899983 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:55.899999 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:55.900691 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:55.900695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:55.900811 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:55.901348 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:55.901364 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:55.904669 (+ 3305us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":276,"threads_started":1}
W20260925 05:01:55.905097 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:55.896210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:44003 (local address 127.30.247.62:37869)
0925 05:01:55.896376 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:55.896381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:55.896665 (+ 284us) server_negotiation.cc:407] Connection header received
0925 05:01:55.896792 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:55.896796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:55.896865 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:55.896949 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:55.897674 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:55.898441 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:55.900128 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:55.900525 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:55.900653 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:55.904856 (+ 4203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.1:44003: BlockingRecv error: recv got EOF from 127.30.247.1:44003 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 05:01:56 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:56 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:56.227061 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:56.217907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.3:53447)
0925 05:01:56.218452 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:56.218468 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:56.218575 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:56.218880 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:56.218884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:56.219280 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:56.219507 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:56.219517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:56.220624 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:56.220627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:56.221629 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:56.221647 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:56.222348 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:56.222352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:56.222469 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:56.223063 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:56.223080 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:56.226730 (+ 3650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":285,"threads_started":1}
W20260925 05:01:56.227248 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:56.217984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:53447 (local address 127.30.247.62:37869)
0925 05:01:56.218142 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:56.218147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:56.218537 (+ 390us) server_negotiation.cc:407] Connection header received
0925 05:01:56.218672 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:56.218676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:56.218732 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:56.218818 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:56.219629 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:56.220415 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:56.221752 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:56.222158 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:56.222248 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:56.227017 (+ 4769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.3:53447: BlockingRecv error: recv got EOF from 127.30.247.3:53447 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 05:01:56 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:56 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:56.556599 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:56.547619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.2:33749)
0925 05:01:56.548108 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:56.548123 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:56.548238 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:56.548646 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:56.548650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:56.549006 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:56.549160 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:56.549167 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:56.550254 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:56.550257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:56.551168 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:56.551178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:56.551873 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:56.551876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:56.551991 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:56.552656 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:56.552690 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:56.556247 (+ 3557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":272,"threads_started":1}
W20260925 05:01:56.556705 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:56.547742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:33749 (local address 127.30.247.62:37869)
0925 05:01:56.547915 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:56.547919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:56.548213 (+ 294us) server_negotiation.cc:407] Connection header received
0925 05:01:56.548367 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:56.548372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:56.548450 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:56.548618 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:56.549333 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:56.550130 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:56.551298 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:56.551755 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:56.551849 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:56.556426 (+ 4577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.2:33749: BlockingRecv error: recv got EOF from 127.30.247.2:33749 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 05:01:56 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:56 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:56.917039 8541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:56.906350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.1:35143)
0925 05:01:56.907049 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:56.907071 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:56.907208 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:56.907793 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:56.907799 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:56.908327 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:56.908648 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:56.908659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:56.910213 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:56.910220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:56.911147 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:56.911158 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:56.912221 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:56.912229 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:56.912440 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:56.913123 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:56.913170 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:56.916798 (+ 3628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":418,"threads_started":1}
W20260925 05:01:56.917238 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:56.906407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:35143 (local address 127.30.247.62:37869)
0925 05:01:56.906656 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:56.906664 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:56.907164 (+ 500us) server_negotiation.cc:407] Connection header received
0925 05:01:56.907523 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:56.907542 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:56.907624 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:56.907747 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:56.908783 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:56.910062 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:56.911291 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:56.912036 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:56.912191 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:56.916988 (+ 4797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.1:35143: BlockingRecv error: recv got EOF from 127.30.247.1:35143 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 05:01:57 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:57 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.3@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:57.237481 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:57.228160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.3:58649)
0925 05:01:57.228672 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:57.228688 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:57.228799 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:57.229131 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:57.229134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:57.229567 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:57.229728 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:57.229737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:57.230900 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:57.230905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:57.232095 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:57.232129 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:57.233146 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:57.233150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:57.233278 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:57.233862 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:57.233879 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:57.237255 (+ 3376us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":267,"threads_started":1}
W20260925 05:01:57.237725 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:57.228197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:58649 (local address 127.30.247.62:37869)
0925 05:01:57.228448 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:57.228452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:57.228767 (+ 315us) server_negotiation.cc:407] Connection header received
0925 05:01:57.228898 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:57.228902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:57.228965 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:57.229047 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:57.229883 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:57.230758 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:57.232253 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:57.232966 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:57.233090 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:57.237448 (+ 4358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.3:58649: BlockingRecv error: recv got EOF from 127.30.247.3:58649 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Sep 25 05:01:57 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:57 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.2@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:57.566668 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:57.557884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.2:34657)
0925 05:01:57.558420 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:57.558440 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:57.558578 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:57.558999 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:57.559003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:57.559411 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:57.559588 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:57.559596 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:57.560612 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:57.560617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:57.561634 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:57.561649 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:57.562342 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:57.562346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:57.562469 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:57.563074 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:57.563091 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:57.566412 (+ 3321us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":257,"threads_started":1}
W20260925 05:01:57.566866 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:57.557940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:34657 (local address 127.30.247.62:37869)
0925 05:01:57.558191 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:57.558200 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:57.558523 (+ 323us) server_negotiation.cc:407] Connection header received
0925 05:01:57.558728 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:57.558736 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:57.558819 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:57.558947 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:57.559713 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:57.560455 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:57.561792 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:57.562219 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:57.562316 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:57.566624 (+ 4308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.2:34657: BlockingRecv error: recv got EOF from 127.30.247.2:34657 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 05:01:57 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:01:57 dist-test-slave-5n86 krb5kdc[7496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.247.1@KRBTEST.COM for oryx/127.30.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:01:57.927030 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:57.917977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37869 (local address 127.30.247.1:57755)
0925 05:01:57.918508 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:57.918524 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:57.918631 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:57.918904 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:57.918907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:57.919367 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:57.919532 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:57.919540 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:57.920606 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:57.920614 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:57.921665 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:57.921678 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:57.922483 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:57.922489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:57.922668 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:57.923380 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:01:57.923404 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:57.926698 (+ 3294us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.247.62:37869: Server oryx/127.30.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":308,"threads_started":1}
W20260925 05:01:57.927135 8233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:01:57.918017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:57755 (local address 127.30.247.62:37869)
0925 05:01:57.918174 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:57.918178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:57.918591 (+ 413us) server_negotiation.cc:407] Connection header received
0925 05:01:57.918719 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:57.918723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:57.918787 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:57.918865 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:57.919658 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:57.920449 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:57.921797 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:57.922341 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:57.922474 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:57.926902 (+ 4428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.247.1:57755: BlockingRecv error: recv got EOF from 127.30.247.1:57755 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 05:01:57.968408 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 8083
I20260925 05:01:57.979987 8546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:57.969377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55880 (local address 127.30.247.1:38189)
0925 05:01:57.969978 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:57.969985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:57.970013 (+ 28us) server_negotiation.cc:407] Connection header received
0925 05:01:57.970068 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:57.970075 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:57.970306 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:57.970473 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:57.971814 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:57.972946 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:57.974042 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:57.974566 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:57.974662 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:57.976994 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:57.977002 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:57.977015 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:57.977046 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:57.979481 (+ 2435us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.30.247.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 05:01:57.979731 (+ 250us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55880: Request ticket server kudu/127.30.247.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":278,"threads_started":1}
W20260925 05:01:57.980084 8546 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55880: Request ticket server kudu/127.30.247.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 05:01:57.980130 31708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 8083: Not authorized: Client connection negotiation failed: client connection to 127.30.247.1:38189: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.30.247.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 05:01:57.980367 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 8083
I20260925 05:01:57.996721 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 8234
I20260925 05:01:58.008481 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:57.997741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51432 (local address 127.30.247.2:36601)
0925 05:01:57.998367 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:57.998374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:57.998401 (+ 27us) server_negotiation.cc:407] Connection header received
0925 05:01:57.998458 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:57.998465 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:57.998698 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:57.998901 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:58.000111 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:58.001272 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:58.002440 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:58.002981 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:58.003121 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:58.005515 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:58.005524 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:58.005538 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:58.005574 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:58.007828 (+ 2254us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.30.247.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 05:01:58.008158 (+ 330us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51432: Request ticket server kudu/127.30.247.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":331,"threads_started":1}
W20260925 05:01:58.008486 31708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 8234: Not authorized: Client connection negotiation failed: client connection to 127.30.247.2:36601: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.30.247.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 05:01:58.008654 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 8234
I20260925 05:01:58.025079 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 8367
I20260925 05:01:58.036697 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:58.026090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49048 (local address 127.30.247.3:37565)
0925 05:01:58.026634 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:58.026641 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:58.026664 (+ 23us) server_negotiation.cc:407] Connection header received
0925 05:01:58.026727 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:58.026733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:58.026962 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:58.027168 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:58.028399 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:58.029478 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:58.030631 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:58.031179 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:58.031329 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:58.033727 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:58.033735 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:58.033748 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:01:58.033784 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:58.036031 (+ 2247us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.30.247.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 05:01:58.036363 (+ 332us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49048: Request ticket server kudu/127.30.247.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":265,"threads_started":1}
W20260925 05:01:58.036690 31708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 8367: Not authorized: Client connection negotiation failed: client connection to 127.30.247.3:37565: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.30.247.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 05:01:58.036778 8548 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49048: Request ticket server kudu/127.30.247.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 05:01:58.036875 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 8367
I20260925 05:01:58.049364 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 8014
I20260925 05:01:58.050499 8076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:01:58.163113 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 8014
2026-09-25T05:01:58Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21272 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T05:01:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:01:58Z Disabled control of system clock
I20260925 05:01:58.236450 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:35575
--webserver_interface=127.30.247.62
--webserver_port=0
--builtin_ntp_servers=127.30.247.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:35575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 05:01:58.544836 8553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:58.545090 8553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:58.545166 8553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:58.551994 8553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:01:58.552086 8553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:58.552122 8553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:58.552175 8553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:01:58.552219 8553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:01:58.560742 8553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:35575
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:35575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:58.561856 8553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:58.563560 8553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:58.575002 8559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:58.575059 8558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:58.575037 8561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:58.575299 8553 server_base.cc:1034] running on GCE node
I20260925 05:01:58.576354 8553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:58.577679 8553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:58.578928 8553 hybrid_clock.cc:648] HybridClock initialized: now 1790312518578879 us; error 39 us; skew 500 ppm
I20260925 05:01:58.579303 8553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:01:58.581887 8553 webserver.cc:460] Webserver started at http://127.30.247.62:44669/ using document root <none> and password file <none>
I20260925 05:01:58.582618 8553 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:58.582783 8553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:58.583209 8553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:58.586515 8553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "727ffc7861c04b14a67c275afb22a952"
format_stamp: "Formatted at 2026-09-25 05:01:58 on dist-test-slave-5n86"
server_key: "4d036e91a097f57e1089b3ce2b859f9a"
server_key_iv: "cdc29ddf94dfffffd12963fe414fed27"
server_key_version: "encryptionkey@0"
I20260925 05:01:58.587276 8553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "727ffc7861c04b14a67c275afb22a952"
format_stamp: "Formatted at 2026-09-25 05:01:58 on dist-test-slave-5n86"
server_key: "4d036e91a097f57e1089b3ce2b859f9a"
server_key_iv: "cdc29ddf94dfffffd12963fe414fed27"
server_key_version: "encryptionkey@0"
I20260925 05:01:58.592087 8553 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 05:01:58.595353 8567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:58.596652 8553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:01:58.596827 8553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "727ffc7861c04b14a67c275afb22a952"
format_stamp: "Formatted at 2026-09-25 05:01:58 on dist-test-slave-5n86"
server_key: "4d036e91a097f57e1089b3ce2b859f9a"
server_key_iv: "cdc29ddf94dfffffd12963fe414fed27"
server_key_version: "encryptionkey@0"
I20260925 05:01:58.597031 8553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:58.609284 8553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:58.609997 8553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:58.610280 8553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:58.627978 8553 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:35575
I20260925 05:01:58.627996 8618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:35575 every 8 connection(s)
I20260925 05:01:58.629221 8553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:01:58.632746 8619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:01:58.633502 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 8553
I20260925 05:01:58.633706 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:01:58.634480 31708 external_mini_cluster.cc:1428] Setting key 672944bb8abddf543aa399e401afb5b0
I20260925 05:01:58.644769 8619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 727ffc7861c04b14a67c275afb22a952: Bootstrap starting.
I20260925 05:01:58.647085 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:58.636454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57860 (local address 127.30.247.62:35575)
0925 05:01:58.637402 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:58.637411 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:58.637451 (+ 40us) server_negotiation.cc:407] Connection header received
0925 05:01:58.639015 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:58.639036 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:58.639458 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:58.639829 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:58.640415 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:58.641662 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:58.642963 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:58.643593 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:58.643792 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:58.645002 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:58.645010 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:58.645022 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:01:58.645523 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:58.645591 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:01:58.645611 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:58.645747 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:58.645966 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 05:01:58.646244 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":316,"threads_started":1}
I20260925 05:01:58.647993 8619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 727ffc7861c04b14a67c275afb22a952: Neither blocks nor log segments found. Creating new log.
I20260925 05:01:58.649124 8619 log.cc:824] T 00000000000000000000000000000000 P 727ffc7861c04b14a67c275afb22a952: Log is configured to *not* fsync() on all Append() calls
I20260925 05:01:58.651628 8619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 727ffc7861c04b14a67c275afb22a952: No bootstrap required, opened a new log
I20260925 05:01:58.658582 8619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 727ffc7861c04b14a67c275afb22a952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727ffc7861c04b14a67c275afb22a952" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35575 } }
I20260925 05:01:58.658895 8619 raft_consensus.cc:374] T 00000000000000000000000000000000 P 727ffc7861c04b14a67c275afb22a952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:01:58.659013 8619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 727ffc7861c04b14a67c275afb22a952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 727ffc7861c04b14a67c275afb22a952, State: Initialized, Role: FOLLOWER
I20260925 05:01:58.659580 8619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 727ffc7861c04b14a67c275afb22a952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727ffc7861c04b14a67c275afb22a952" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35575 } }
I20260925 05:01:58.659797 8619 raft_consensus.cc:388] T 00000000000000000000000000000000 P 727ffc7861c04b14a67c275afb22a952 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:01:58.659948 8619 raft_consensus.cc:482] T 00000000000000000000000000000000 P 727ffc7861c04b14a67c275afb22a952 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:01:58.660127 8619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 727ffc7861c04b14a67c275afb22a952 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:01:58.662437 8619 raft_consensus.cc:504] T 00000000000000000000000000000000 P 727ffc7861c04b14a67c275afb22a952 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727ffc7861c04b14a67c275afb22a952" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35575 } }
I20260925 05:01:58.662895 8619 leader_election.cc:302] T 00000000000000000000000000000000 P 727ffc7861c04b14a67c275afb22a952 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 727ffc7861c04b14a67c275afb22a952; no voters:
I20260925 05:01:58.663435 8619 leader_election.cc:288] T 00000000000000000000000000000000 P 727ffc7861c04b14a67c275afb22a952 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:01:58.663576 8624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 727ffc7861c04b14a67c275afb22a952 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:01:58.664324 8624 raft_consensus.cc:686] T 00000000000000000000000000000000 P 727ffc7861c04b14a67c275afb22a952 [term 1 LEADER]: Becoming Leader. State: Replica: 727ffc7861c04b14a67c275afb22a952, State: Running, Role: LEADER
I20260925 05:01:58.664837 8624 consensus_queue.cc:237] T 00000000000000000000000000000000 P 727ffc7861c04b14a67c275afb22a952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727ffc7861c04b14a67c275afb22a952" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35575 } }
I20260925 05:01:58.665460 8619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 727ffc7861c04b14a67c275afb22a952 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:01:58.667271 8625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 727ffc7861c04b14a67c275afb22a952 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "727ffc7861c04b14a67c275afb22a952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727ffc7861c04b14a67c275afb22a952" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35575 } } }
I20260925 05:01:58.667622 8625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 727ffc7861c04b14a67c275afb22a952 [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:58.668160 8626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 727ffc7861c04b14a67c275afb22a952 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 727ffc7861c04b14a67c275afb22a952. Latest consensus state: current_term: 1 leader_uuid: "727ffc7861c04b14a67c275afb22a952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727ffc7861c04b14a67c275afb22a952" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35575 } } }
I20260925 05:01:58.668453 8626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 727ffc7861c04b14a67c275afb22a952 [sys.catalog]: This master's current role is: LEADER
I20260925 05:01:58.669544 8629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:01:58.674256 8629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:01:58.682081 8629 catalog_manager.cc:1347] Generated new cluster ID: 228c1899b7cf47ba8ca95eb2c5c6827e
I20260925 05:01:58.682173 8629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:01:58.695281 8629 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:01:58.696642 8629 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:01:58.715289 8629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 727ffc7861c04b14a67c275afb22a952: Generated new TSK 0
I20260925 05:01:58.716092 8629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:01:58.721530 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:0
--local_ip_for_outbound_sockets=127.30.247.1
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:35575
--builtin_ntp_servers=127.30.247.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 05:01:59.060350 8643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:59.060665 8643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:59.060820 8643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:59.068071 8643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:59.068156 8643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:59.068307 8643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:01:59.077283 8643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:35575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:59.078480 8643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:59.080173 8643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:59.092938 8648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:59.093024 8649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:59.092900 8651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:59.093990 8643 server_base.cc:1034] running on GCE node
I20260925 05:01:59.094492 8643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:59.095459 8643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:59.096629 8643 hybrid_clock.cc:648] HybridClock initialized: now 1790312519096567 us; error 67 us; skew 500 ppm
I20260925 05:01:59.096948 8643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:01:59.103390 8643 webserver.cc:460] Webserver started at http://127.30.247.1:40427/ using document root <none> and password file <none>
I20260925 05:01:59.104080 8643 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:59.104205 8643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:59.104592 8643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:59.107337 8643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/data/instance:
uuid: "fadd004ea63b413f93531f634ecf1850"
format_stamp: "Formatted at 2026-09-25 05:01:59 on dist-test-slave-5n86"
server_key: "f70f98f5824feea23e7e207a549c7c25"
server_key_iv: "74f7b83d2c016868507014d16bf25541"
server_key_version: "encryptionkey@0"
I20260925 05:01:59.108086 8643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance:
uuid: "fadd004ea63b413f93531f634ecf1850"
format_stamp: "Formatted at 2026-09-25 05:01:59 on dist-test-slave-5n86"
server_key: "f70f98f5824feea23e7e207a549c7c25"
server_key_iv: "74f7b83d2c016868507014d16bf25541"
server_key_version: "encryptionkey@0"
I20260925 05:01:59.112951 8643 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 05:01:59.116045 8657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:59.117269 8643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 05:01:59.117436 8643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "fadd004ea63b413f93531f634ecf1850"
format_stamp: "Formatted at 2026-09-25 05:01:59 on dist-test-slave-5n86"
server_key: "f70f98f5824feea23e7e207a549c7c25"
server_key_iv: "74f7b83d2c016868507014d16bf25541"
server_key_version: "encryptionkey@0"
I20260925 05:01:59.117632 8643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:59.137066 8643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:59.137785 8643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:59.138068 8643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:59.138922 8643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:59.140229 8643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:59.140358 8643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:59.140475 8643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:59.140580 8643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:59.172525 8643 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:40385
I20260925 05:01:59.172647 8769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:40385 every 8 connection(s)
I20260925 05:01:59.173785 8643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:01:59.174755 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 8643
I20260925 05:01:59.174947 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance
I20260925 05:01:59.175854 31708 external_mini_cluster.cc:1428] Setting key dd25b2dfa865c48814540a507eb6560f
I20260925 05:01:59.178941 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:0
--local_ip_for_outbound_sockets=127.30.247.2
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:35575
--builtin_ntp_servers=127.30.247.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 05:01:59.189098 8772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:59.176488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:39381 (local address 127.30.247.62:35575)
0925 05:01:59.176889 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:59.176893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:59.178334 (+ 1441us) server_negotiation.cc:407] Connection header received
0925 05:01:59.181070 (+ 2736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:59.181077 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:59.181164 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:59.181267 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:59.182645 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:59.183644 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:59.186106 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:59.186773 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:59.186909 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:59.188389 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:59.188393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:59.188397 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:01:59.188412 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:59.188450 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:01:59.188462 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:59.188528 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:59.188790 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 05:01:59.188861 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":208,"threads_started":1}
I20260925 05:01:59.190448 8773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:59.177154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:35575 (local address 127.30.247.1:39381)
0925 05:01:59.178174 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:59.178219 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:59.180774 (+ 2555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:59.181515 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:59.181527 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:59.181637 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:59.182413 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:59.182440 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:59.183800 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:59.183808 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:59.185970 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:59.185989 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:59.187008 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:59.187016 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:59.187335 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:59.188057 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:01:59.188063 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:59.188112 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:01:59.188209 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:59.188574 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:59.188586 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:59.188597 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 05:01:59.188770 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 05:01:59.189179 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":272,"threads_started":1}
I20260925 05:01:59.191963 8770 heartbeater.cc:347] Connected to a master server at 127.30.247.62:35575
I20260925 05:01:59.192265 8770 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:59.193140 8770 heartbeater.cc:511] Master 127.30.247.62:35575 requested a full tablet report, sending...
I20260925 05:01:59.195287 8584 ts_manager.cc:194] Registered new tserver with Master: fadd004ea63b413f93531f634ecf1850 (127.30.247.1:40385)
I20260925 05:01:59.197553 8584 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.247.1:39381
W20260925 05:01:59.492779 8774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:59.493029 8774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:59.493106 8774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:59.499979 8774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:59.500063 8774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:59.500159 8774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:01:59.508894 8774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:35575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:59.509979 8774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:59.511548 8774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:59.523713 8779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:59.523659 8780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:59.523744 8782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:59.524446 8774 server_base.cc:1034] running on GCE node
I20260925 05:01:59.525017 8774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:59.525823 8774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:59.527022 8774 hybrid_clock.cc:648] HybridClock initialized: now 1790312519526983 us; error 32 us; skew 500 ppm
I20260925 05:01:59.527345 8774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:01:59.529574 8774 webserver.cc:460] Webserver started at http://127.30.247.2:41809/ using document root <none> and password file <none>
I20260925 05:01:59.530229 8774 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:59.530339 8774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:59.530655 8774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:59.533366 8774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/data/instance:
uuid: "24d654f609664bb49b1d67eb1e42d86e"
format_stamp: "Formatted at 2026-09-25 05:01:59 on dist-test-slave-5n86"
server_key: "9938434a0fd36ac58b53e7eff1b5f6cd"
server_key_iv: "8dd93b75bb66462b39284a32105cbec5"
server_key_version: "encryptionkey@0"
I20260925 05:01:59.534112 8774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance:
uuid: "24d654f609664bb49b1d67eb1e42d86e"
format_stamp: "Formatted at 2026-09-25 05:01:59 on dist-test-slave-5n86"
server_key: "9938434a0fd36ac58b53e7eff1b5f6cd"
server_key_iv: "8dd93b75bb66462b39284a32105cbec5"
server_key_version: "encryptionkey@0"
I20260925 05:01:59.538910 8774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 05:01:59.542178 8788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:59.543438 8774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:01:59.543608 8774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "24d654f609664bb49b1d67eb1e42d86e"
format_stamp: "Formatted at 2026-09-25 05:01:59 on dist-test-slave-5n86"
server_key: "9938434a0fd36ac58b53e7eff1b5f6cd"
server_key_iv: "8dd93b75bb66462b39284a32105cbec5"
server_key_version: "encryptionkey@0"
I20260925 05:01:59.543817 8774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:59.553701 8774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:59.554406 8774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:59.554689 8774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:59.555540 8774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:59.556905 8774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:59.557027 8774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:59.557154 8774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:59.557243 8774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:59.588763 8774 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:42993
I20260925 05:01:59.588838 8900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:42993 every 8 connection(s)
I20260925 05:01:59.589973 8774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:01:59.592868 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 8774
I20260925 05:01:59.593029 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance
I20260925 05:01:59.593638 31708 external_mini_cluster.cc:1428] Setting key b312696025f940efa179cdc5db9fdce7
I20260925 05:01:59.596184 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:0
--local_ip_for_outbound_sockets=127.30.247.3
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:35575
--builtin_ntp_servers=127.30.247.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 05:01:59.602128 8772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:59.592294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:33045 (local address 127.30.247.62:35575)
0925 05:01:59.592458 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 05:01:59.592463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:01:59.594065 (+ 1602us) server_negotiation.cc:407] Connection header received
0925 05:01:59.595931 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:01:59.595937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:01:59.596013 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:01:59.596128 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:01:59.597247 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:59.597973 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:59.599453 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:01:59.599837 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:01:59.599967 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:59.601352 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:01:59.601357 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:01:59.601362 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:01:59.601387 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:01:59.601415 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:01:59.601427 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:01:59.601519 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 05:01:59.601733 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 05:01:59.601828 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 05:01:59.603181 8903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:01:59.592973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:35575 (local address 127.30.247.2:33045)
0925 05:01:59.593925 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0925 05:01:59.593958 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 05:01:59.595681 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:01:59.596371 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:01:59.596385 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:01:59.596499 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:01:59.597088 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:59.597104 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:59.598111 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:59.598117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:59.599301 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:01:59.599312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:01:59.600098 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:01:59.600110 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:01:59.600305 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:01:59.601010 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:01:59.601012 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:01:59.601047 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:01:59.601158 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:01:59.601517 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:01:59.601528 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:01:59.601538 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 05:01:59.601707 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 05:01:59.602077 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":286,"threads_started":1}
I20260925 05:01:59.604524 8901 heartbeater.cc:347] Connected to a master server at 127.30.247.62:35575
I20260925 05:01:59.604851 8901 heartbeater.cc:464] Registering TS with master...
I20260925 05:01:59.605659 8901 heartbeater.cc:511] Master 127.30.247.62:35575 requested a full tablet report, sending...
I20260925 05:01:59.607009 8584 ts_manager.cc:194] Registered new tserver with Master: 24d654f609664bb49b1d67eb1e42d86e (127.30.247.2:42993)
I20260925 05:01:59.607780 8584 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.247.2:33045
W20260925 05:01:59.907214 8904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:01:59.907510 8904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:01:59.907684 8904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:01:59.914620 8904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:01:59.914700 8904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:01:59.914880 8904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:01:59.923492 8904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:35575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:01:59.924697 8904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:01:59.926373 8904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:01:59.938704 8909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:59.938789 8910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:01:59.938809 8912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:01:59.939459 8904 server_base.cc:1034] running on GCE node
I20260925 05:01:59.940019 8904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:01:59.940987 8904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:01:59.942183 8904 hybrid_clock.cc:648] HybridClock initialized: now 1790312519942128 us; error 48 us; skew 500 ppm
I20260925 05:01:59.942480 8904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:01:59.944578 8904 webserver.cc:460] Webserver started at http://127.30.247.3:45777/ using document root <none> and password file <none>
I20260925 05:01:59.945248 8904 fs_manager.cc:359] Metadata directory not provided
I20260925 05:01:59.945362 8904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:01:59.945686 8904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:01:59.948570 8904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/data/instance:
uuid: "a07a1916f49b499290ca0c301b0fc7d8"
format_stamp: "Formatted at 2026-09-25 05:01:59 on dist-test-slave-5n86"
server_key: "cda4f3213e031cd74a855a3a7f9acfb5"
server_key_iv: "98868bf948813f7e9ef6cda02745d9ee"
server_key_version: "encryptionkey@0"
I20260925 05:01:59.949380 8904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance:
uuid: "a07a1916f49b499290ca0c301b0fc7d8"
format_stamp: "Formatted at 2026-09-25 05:01:59 on dist-test-slave-5n86"
server_key: "cda4f3213e031cd74a855a3a7f9acfb5"
server_key_iv: "98868bf948813f7e9ef6cda02745d9ee"
server_key_version: "encryptionkey@0"
I20260925 05:01:59.954105 8904 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 05:01:59.957538 8918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:59.958940 8904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:01:59.959110 8904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "a07a1916f49b499290ca0c301b0fc7d8"
format_stamp: "Formatted at 2026-09-25 05:01:59 on dist-test-slave-5n86"
server_key: "cda4f3213e031cd74a855a3a7f9acfb5"
server_key_iv: "98868bf948813f7e9ef6cda02745d9ee"
server_key_version: "encryptionkey@0"
I20260925 05:01:59.959389 8904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:01:59.980273 8904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:01:59.981051 8904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:01:59.981336 8904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:01:59.982191 8904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:01:59.983474 8904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:01:59.983601 8904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:01:59.983716 8904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:01:59.983799 8904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:00.015579 8904 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:46257
I20260925 05:02:00.015667 9030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:46257 every 8 connection(s)
I20260925 05:02:00.016824 8904 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:02:00.018191 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 8904
I20260925 05:02:00.018368 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance
I20260925 05:02:00.018889 31708 external_mini_cluster.cc:1428] Setting key e78ed90b142936fd60af701055b0e59f
I20260925 05:02:00.030422 8772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:00.021176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:38637 (local address 127.30.247.62:35575)
0925 05:02:00.021387 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:00.021393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:00.022910 (+ 1517us) server_negotiation.cc:407] Connection header received
0925 05:02:00.024479 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:00.024483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:00.024598 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:00.024697 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:00.025618 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:00.026347 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:00.027968 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:00.028350 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:00.028445 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:00.029695 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:02:00.029700 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:02:00.029703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:02:00.029718 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:02:00.029759 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:02:00.029771 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:02:00.029850 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:00.030112 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 05:02:00.030184 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:02:00.031359 9033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:00.021887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:35575 (local address 127.30.247.3:38637)
0925 05:02:00.022773 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0925 05:02:00.022810 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:02:00.024255 (+ 1445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:02:00.024855 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:02:00.024862 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:02:00.024939 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:02:00.025478 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:00.025493 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:00.026504 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:00.026512 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:00.027810 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:00.027826 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:00.028605 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:00.028611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:00.028805 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:00.029394 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:02:00.029397 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:02:00.029432 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:02:00.029504 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:02:00.029888 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:02:00.029896 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:02:00.029902 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 05:02:00.030053 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 05:02:00.030323 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":288,"threads_started":1}
I20260925 05:02:00.032819 9031 heartbeater.cc:347] Connected to a master server at 127.30.247.62:35575
I20260925 05:02:00.033135 9031 heartbeater.cc:464] Registering TS with master...
I20260925 05:02:00.033911 9031 heartbeater.cc:511] Master 127.30.247.62:35575 requested a full tablet report, sending...
I20260925 05:02:00.035250 8584 ts_manager.cc:194] Registered new tserver with Master: a07a1916f49b499290ca0c301b0fc7d8 (127.30.247.3:46257)
I20260925 05:02:00.035977 8584 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.247.3:38637
I20260925 05:02:00.046384 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 05:02:00.052623 8772 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:02:00.050866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55810 (local address 127.30.247.62:35575)
0925 05:02:00.051036 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:00.051041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:00.051538 (+ 497us) server_negotiation.cc:407] Connection header received
0925 05:02:00.051744 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:00.051748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:00.051817 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:00.051920 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:00.052321 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55810: BlockingRecv error: recv got EOF from 127.0.0.1:55810 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 05:02:00.053388 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 8643
I20260925 05:02:00.060655 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:00.054232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58360 (local address 127.30.247.1:40385)
0925 05:02:00.054735 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:00.054741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:00.054766 (+ 25us) server_negotiation.cc:407] Connection header received
0925 05:02:00.054827 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:00.054833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:00.055029 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:00.055152 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:00.055476 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:00.056455 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:00.057731 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:00.058290 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:00.058392 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:00.059341 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:02:00.059349 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:02:00.059360 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:02:00.059918 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:02:00.059960 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:02:00.059978 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:02:00.060063 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:00.060197 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 05:02:00.060308 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":254,"threads_started":1}
I20260925 05:02:00.061869 8765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:02:00.143277 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 8643
I20260925 05:02:00.158977 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 8774
I20260925 05:02:00.166723 9042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:00.159811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48126 (local address 127.30.247.2:42993)
0925 05:02:00.160366 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:00.160373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:00.160401 (+ 28us) server_negotiation.cc:407] Connection header received
0925 05:02:00.160468 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:00.160474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:00.160832 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:00.160962 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:00.161282 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:00.162261 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:00.163477 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:00.164199 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:00.164295 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:00.165318 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:02:00.165325 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:02:00.165338 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:02:00.165943 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:02:00.165986 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:02:00.166005 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:02:00.166097 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:00.166256 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 05:02:00.166367 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":281,"threads_started":1}
I20260925 05:02:00.167896 8896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:02:00.248589 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 8774
I20260925 05:02:00.264285 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 8904
I20260925 05:02:00.272182 9044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:00.265255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38002 (local address 127.30.247.3:46257)
0925 05:02:00.265833 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:00.265840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:00.265909 (+ 69us) server_negotiation.cc:407] Connection header received
0925 05:02:00.265984 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:00.265990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:00.266199 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:00.266326 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:00.266737 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:00.267842 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:00.269126 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:00.269726 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:00.269827 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:00.270868 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:02:00.270876 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:02:00.270888 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:02:00.271484 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:02:00.271527 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:02:00.271551 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:02:00.271634 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:00.271794 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 05:02:00.271901 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":268,"threads_started":1}
I20260925 05:02:00.273445 9026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:02:00.355011 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 8904
I20260925 05:02:00.371018 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 8553
I20260925 05:02:00.372107 8614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:02:00.470734 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 8553
2026-09-25T05:02:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2302 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T05:02:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:02:00Z Disabled control of system clock
I20260925 05:02:00.536474 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:39983
--webserver_interface=127.30.247.62
--webserver_port=0
--builtin_ntp_servers=127.30.247.20:43663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:39983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 05:02:00.840777 9050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:02:00.841023 9050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:02:00.841194 9050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:02:00.848009 9050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:02:00.848095 9050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:02:00.848131 9050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:02:00.848203 9050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:02:00.848268 9050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:02:00.856969 9050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:43663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:39983
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:39983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:02:00.858182 9050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:02:00.859947 9050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:02:00.871601 9055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:02:00.871634 9056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:02:00.871564 9058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:02:00.872352 9050 server_base.cc:1034] running on GCE node
I20260925 05:02:00.873199 9050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:02:00.874465 9050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:02:00.875651 9050 hybrid_clock.cc:648] HybridClock initialized: now 1790312520875602 us; error 52 us; skew 500 ppm
I20260925 05:02:00.875937 9050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:02:00.878101 9050 webserver.cc:460] Webserver started at http://127.30.247.62:43967/ using document root <none> and password file <none>
I20260925 05:02:00.878698 9050 fs_manager.cc:359] Metadata directory not provided
I20260925 05:02:00.878785 9050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:02:00.879043 9050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:02:00.881733 9050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "5b2ec74b08a5442cb079a6338f9aa8bb"
format_stamp: "Formatted at 2026-09-25 05:02:00 on dist-test-slave-5n86"
server_key: "b1d311861b775eb7915ecd37fdabd8ac"
server_key_iv: "2d7550e56732c35fd375f1d94e40c9ff"
server_key_version: "encryptionkey@0"
I20260925 05:02:00.882510 9050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "5b2ec74b08a5442cb079a6338f9aa8bb"
format_stamp: "Formatted at 2026-09-25 05:02:00 on dist-test-slave-5n86"
server_key: "b1d311861b775eb7915ecd37fdabd8ac"
server_key_iv: "2d7550e56732c35fd375f1d94e40c9ff"
server_key_version: "encryptionkey@0"
I20260925 05:02:00.887192 9050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 05:02:00.890384 9064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:00.891609 9050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:02:00.891784 9050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "5b2ec74b08a5442cb079a6338f9aa8bb"
format_stamp: "Formatted at 2026-09-25 05:02:00 on dist-test-slave-5n86"
server_key: "b1d311861b775eb7915ecd37fdabd8ac"
server_key_iv: "2d7550e56732c35fd375f1d94e40c9ff"
server_key_version: "encryptionkey@0"
I20260925 05:02:00.891985 9050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:02:00.897380 9050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:02:00.898074 9050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:02:00.898365 9050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:02:00.915696 9050 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:39983
I20260925 05:02:00.915732 9115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:39983 every 8 connection(s)
I20260925 05:02:00.916999 9050 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:02:00.920323 9116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:02:00.922838 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 9050
I20260925 05:02:00.923074 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:02:00.923661 31708 external_mini_cluster.cc:1428] Setting key 9bf93bac315d749dbb74e71dd781f286
I20260925 05:02:00.931962 9119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:00.925974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44718 (local address 127.30.247.62:39983)
0925 05:02:00.926888 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:00.926899 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:00.926936 (+ 37us) server_negotiation.cc:407] Connection header received
0925 05:02:00.928491 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:00.928511 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:00.928967 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:00.929369 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:00.930435 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:02:00.930441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:02:00.930456 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:02:00.930469 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:02:00.930519 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:02:00.930534 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:02:00.930761 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:00.930902 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 05:02:00.931123 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":331,"threads_started":1}
I20260925 05:02:00.932461 9116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b2ec74b08a5442cb079a6338f9aa8bb: Bootstrap starting.
I20260925 05:02:00.935482 9116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b2ec74b08a5442cb079a6338f9aa8bb: Neither blocks nor log segments found. Creating new log.
I20260925 05:02:00.936506 9116 log.cc:824] T 00000000000000000000000000000000 P 5b2ec74b08a5442cb079a6338f9aa8bb: Log is configured to *not* fsync() on all Append() calls
I20260925 05:02:00.939397 9116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b2ec74b08a5442cb079a6338f9aa8bb: No bootstrap required, opened a new log
I20260925 05:02:00.946604 9116 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b2ec74b08a5442cb079a6338f9aa8bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2ec74b08a5442cb079a6338f9aa8bb" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 39983 } }
I20260925 05:02:00.946979 9116 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b2ec74b08a5442cb079a6338f9aa8bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:02:00.947086 9116 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b2ec74b08a5442cb079a6338f9aa8bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b2ec74b08a5442cb079a6338f9aa8bb, State: Initialized, Role: FOLLOWER
I20260925 05:02:00.947674 9116 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b2ec74b08a5442cb079a6338f9aa8bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2ec74b08a5442cb079a6338f9aa8bb" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 39983 } }
I20260925 05:02:00.947914 9116 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b2ec74b08a5442cb079a6338f9aa8bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:02:00.948055 9116 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b2ec74b08a5442cb079a6338f9aa8bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:02:00.948230 9116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b2ec74b08a5442cb079a6338f9aa8bb [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:02:00.950815 9116 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b2ec74b08a5442cb079a6338f9aa8bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2ec74b08a5442cb079a6338f9aa8bb" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 39983 } }
I20260925 05:02:00.951320 9116 leader_election.cc:302] T 00000000000000000000000000000000 P 5b2ec74b08a5442cb079a6338f9aa8bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b2ec74b08a5442cb079a6338f9aa8bb; no voters:
I20260925 05:02:00.951949 9116 leader_election.cc:288] T 00000000000000000000000000000000 P 5b2ec74b08a5442cb079a6338f9aa8bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:02:00.952054 9121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b2ec74b08a5442cb079a6338f9aa8bb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:02:00.952886 9121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b2ec74b08a5442cb079a6338f9aa8bb [term 1 LEADER]: Becoming Leader. State: Replica: 5b2ec74b08a5442cb079a6338f9aa8bb, State: Running, Role: LEADER
I20260925 05:02:00.953514 9121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b2ec74b08a5442cb079a6338f9aa8bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2ec74b08a5442cb079a6338f9aa8bb" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 39983 } }
I20260925 05:02:00.954100 9116 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b2ec74b08a5442cb079a6338f9aa8bb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:02:00.956470 9123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b2ec74b08a5442cb079a6338f9aa8bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b2ec74b08a5442cb079a6338f9aa8bb. Latest consensus state: current_term: 1 leader_uuid: "5b2ec74b08a5442cb079a6338f9aa8bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2ec74b08a5442cb079a6338f9aa8bb" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 39983 } } }
I20260925 05:02:00.956650 9122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b2ec74b08a5442cb079a6338f9aa8bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b2ec74b08a5442cb079a6338f9aa8bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2ec74b08a5442cb079a6338f9aa8bb" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 39983 } } }
I20260925 05:02:00.956988 9122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b2ec74b08a5442cb079a6338f9aa8bb [sys.catalog]: This master's current role is: LEADER
I20260925 05:02:00.956964 9123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b2ec74b08a5442cb079a6338f9aa8bb [sys.catalog]: This master's current role is: LEADER
I20260925 05:02:00.961647 9130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:02:00.965786 9130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:02:00.973909 9130 catalog_manager.cc:1347] Generated new cluster ID: dc6eb44b08d04fd6af018c68ed986ecb
I20260925 05:02:00.974009 9130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:02:00.999336 9130 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:02:00.999804 9130 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:02:01.007761 9130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b2ec74b08a5442cb079a6338f9aa8bb: Generated new TSK 0
I20260925 05:02:01.008634 9130 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:02:01.016261 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:0
--local_ip_for_outbound_sockets=127.30.247.1
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:39983
--builtin_ntp_servers=127.30.247.20:43663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 05:02:01.329849 9140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:02:01.330144 9140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:02:01.330299 9140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:02:01.338495 9140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:02:01.338611 9140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:02:01.338778 9140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:02:01.347692 9140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:43663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:39983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:02:01.348922 9140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:02:01.350605 9140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:02:01.363325 9145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:02:01.363261 9146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:02:01.363485 9148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:02:01.364399 9140 server_base.cc:1034] running on GCE node
I20260925 05:02:01.364970 9140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:02:01.365792 9140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:02:01.366999 9140 hybrid_clock.cc:648] HybridClock initialized: now 1790312521366958 us; error 32 us; skew 500 ppm
I20260925 05:02:01.367318 9140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:02:01.369588 9140 webserver.cc:460] Webserver started at http://127.30.247.1:45323/ using document root <none> and password file <none>
I20260925 05:02:01.370234 9140 fs_manager.cc:359] Metadata directory not provided
I20260925 05:02:01.370318 9140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:02:01.370606 9140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:02:01.373318 9140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/data/instance:
uuid: "945db791eb614cea92a4e5f6a9949ed0"
format_stamp: "Formatted at 2026-09-25 05:02:01 on dist-test-slave-5n86"
server_key: "e36811b7e9e59023d7dbc54851659036"
server_key_iv: "34a205b89d1145a56300113e3364a9f1"
server_key_version: "encryptionkey@0"
I20260925 05:02:01.374073 9140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance:
uuid: "945db791eb614cea92a4e5f6a9949ed0"
format_stamp: "Formatted at 2026-09-25 05:02:01 on dist-test-slave-5n86"
server_key: "e36811b7e9e59023d7dbc54851659036"
server_key_iv: "34a205b89d1145a56300113e3364a9f1"
server_key_version: "encryptionkey@0"
I20260925 05:02:01.379060 9140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 05:02:01.382246 9154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:01.383472 9140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260925 05:02:01.383639 9140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "945db791eb614cea92a4e5f6a9949ed0"
format_stamp: "Formatted at 2026-09-25 05:02:01 on dist-test-slave-5n86"
server_key: "e36811b7e9e59023d7dbc54851659036"
server_key_iv: "34a205b89d1145a56300113e3364a9f1"
server_key_version: "encryptionkey@0"
I20260925 05:02:01.383860 9140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:02:01.389209 9140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:02:01.390033 9140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:02:01.390290 9140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:02:01.391131 9140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:02:01.392630 9140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:02:01.392742 9140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:01.392874 9140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:02:01.392961 9140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:01.425033 9140 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:37963
I20260925 05:02:01.425150 9266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:37963 every 8 connection(s)
I20260925 05:02:01.426311 9140 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:02:01.434127 9119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:01.428752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:39067 (local address 127.30.247.62:39983)
0925 05:02:01.428937 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:01.428941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:01.430206 (+ 1265us) server_negotiation.cc:407] Connection header received
0925 05:02:01.431918 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:01.431924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:01.431993 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:01.432112 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:01.433413 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:02:01.433419 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:02:01.433424 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:02:01.433442 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:02:01.433478 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:02:01.433490 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:02:01.433607 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:01.433803 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 05:02:01.433876 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 05:02:01.434995 9269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:01.429182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:39983 (local address 127.30.247.1:39067)
0925 05:02:01.430053 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0925 05:02:01.430086 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 05:02:01.431649 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:02:01.432329 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:02:01.432336 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:02:01.432458 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:02:01.433136 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:02:01.433139 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:02:01.433174 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:02:01.433252 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:02:01.433582 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:02:01.433591 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:02:01.433598 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 05:02:01.433891 (+ 293us) client_negotiation.cc:240] Negotiation successful
0925 05:02:01.434118 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":253,"threads_started":1}
I20260925 05:02:01.435776 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 9140
I20260925 05:02:01.436014 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance
I20260925 05:02:01.436581 9267 heartbeater.cc:347] Connected to a master server at 127.30.247.62:39983
I20260925 05:02:01.436787 31708 external_mini_cluster.cc:1428] Setting key c9423b9dc3cfba09fdf1ef627b4fba1c
I20260925 05:02:01.436911 9267 heartbeater.cc:464] Registering TS with master...
I20260925 05:02:01.437564 9267 heartbeater.cc:511] Master 127.30.247.62:39983 requested a full tablet report, sending...
I20260925 05:02:01.439975 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:0
--local_ip_for_outbound_sockets=127.30.247.2
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:39983
--builtin_ntp_servers=127.30.247.20:43663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 05:02:01.439976 9081 ts_manager.cc:194] Registered new tserver with Master: 945db791eb614cea92a4e5f6a9949ed0 (127.30.247.1:37963)
W20260925 05:02:01.750957 9270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:02:01.751255 9270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:02:01.751415 9270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:02:01.758414 9270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:02:01.758539 9270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:02:01.758719 9270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:02:01.767308 9270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:43663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:39983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:02:01.768476 9270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:02:01.770252 9270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:02:01.782500 9276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:02:01.782444 9278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:02:01.782531 9275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:02:01.783241 9270 server_base.cc:1034] running on GCE node
I20260925 05:02:01.783797 9270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:02:01.784626 9270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:02:01.785837 9270 hybrid_clock.cc:648] HybridClock initialized: now 1790312521785797 us; error 40 us; skew 500 ppm
I20260925 05:02:01.786156 9270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:02:01.792634 9270 webserver.cc:460] Webserver started at http://127.30.247.2:37329/ using document root <none> and password file <none>
I20260925 05:02:01.793246 9270 fs_manager.cc:359] Metadata directory not provided
I20260925 05:02:01.793370 9270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:02:01.793684 9270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:02:01.796391 9270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/data/instance:
uuid: "e17ade80dbb44022985730f5dcabedd2"
format_stamp: "Formatted at 2026-09-25 05:02:01 on dist-test-slave-5n86"
server_key: "f43cc0aacfb6f4c86a25fa4b486c7cab"
server_key_iv: "79cbeca6a9761eb213149323df59d2bb"
server_key_version: "encryptionkey@0"
I20260925 05:02:01.797207 9270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance:
uuid: "e17ade80dbb44022985730f5dcabedd2"
format_stamp: "Formatted at 2026-09-25 05:02:01 on dist-test-slave-5n86"
server_key: "f43cc0aacfb6f4c86a25fa4b486c7cab"
server_key_iv: "79cbeca6a9761eb213149323df59d2bb"
server_key_version: "encryptionkey@0"
I20260925 05:02:01.801798 9270 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 05:02:01.805006 9284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:01.806161 9270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:02:01.806335 9270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "e17ade80dbb44022985730f5dcabedd2"
format_stamp: "Formatted at 2026-09-25 05:02:01 on dist-test-slave-5n86"
server_key: "f43cc0aacfb6f4c86a25fa4b486c7cab"
server_key_iv: "79cbeca6a9761eb213149323df59d2bb"
server_key_version: "encryptionkey@0"
I20260925 05:02:01.806536 9270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:02:01.811885 9270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:02:01.812707 9270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:02:01.812965 9270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:02:01.813793 9270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:02:01.815057 9270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:02:01.815176 9270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:01.815305 9270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:02:01.815392 9270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:01.847064 9270 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:36341
I20260925 05:02:01.847141 9396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:36341 every 8 connection(s)
I20260925 05:02:01.848287 9270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:02:01.849931 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 9270
I20260925 05:02:01.850111 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance
I20260925 05:02:01.850571 31708 external_mini_cluster.cc:1428] Setting key de16ea80e59cdee2400fd06162465681
I20260925 05:02:01.853353 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:0
--local_ip_for_outbound_sockets=127.30.247.3
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:39983
--builtin_ntp_servers=127.30.247.20:43663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 05:02:01.856937 9119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:01.850952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:59191 (local address 127.30.247.62:39983)
0925 05:02:01.851158 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:01.851165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:01.852686 (+ 1521us) server_negotiation.cc:407] Connection header received
0925 05:02:01.854514 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:01.854519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:01.854580 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:01.854695 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:01.856078 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:02:01.856086 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:02:01.856094 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:02:01.856124 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:02:01.856167 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:02:01.856185 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:02:01.856315 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:01.856475 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 05:02:01.856594 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 05:02:01.857689 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:01.851482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:39983 (local address 127.30.247.2:59191)
0925 05:02:01.852508 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0925 05:02:01.852581 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0925 05:02:01.854231 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:02:01.854923 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:02:01.854931 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:02:01.855018 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:02:01.855778 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:02:01.855781 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:02:01.855823 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:02:01.855904 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:02:01.856285 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:02:01.856292 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:02:01.856299 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 05:02:01.856503 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 05:02:01.856754 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":282,"threads_started":1}
I20260925 05:02:01.859062 9397 heartbeater.cc:347] Connected to a master server at 127.30.247.62:39983
I20260925 05:02:01.859385 9397 heartbeater.cc:464] Registering TS with master...
I20260925 05:02:01.860060 9397 heartbeater.cc:511] Master 127.30.247.62:39983 requested a full tablet report, sending...
I20260925 05:02:01.861631 9081 ts_manager.cc:194] Registered new tserver with Master: e17ade80dbb44022985730f5dcabedd2 (127.30.247.2:36341)
W20260925 05:02:02.183452 9400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:02:02.183694 9400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:02:02.183804 9400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:02:02.191165 9400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:02:02.191251 9400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:02:02.191351 9400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:02:02.200325 9400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:43663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:39983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:02:02.201432 9400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:02:02.202966 9400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:02:02.215221 9406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:02:02.215319 9405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:02:02.215183 9408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:02:02.216025 9400 server_base.cc:1034] running on GCE node
I20260925 05:02:02.216497 9400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:02:02.217348 9400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:02:02.218535 9400 hybrid_clock.cc:648] HybridClock initialized: now 1790312522218487 us; error 48 us; skew 500 ppm
I20260925 05:02:02.218827 9400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:02:02.220872 9400 webserver.cc:460] Webserver started at http://127.30.247.3:38139/ using document root <none> and password file <none>
I20260925 05:02:02.221489 9400 fs_manager.cc:359] Metadata directory not provided
I20260925 05:02:02.221596 9400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:02:02.221904 9400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:02:02.224447 9400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/data/instance:
uuid: "4a8bc636783f44d8adaa606389cf35f7"
format_stamp: "Formatted at 2026-09-25 05:02:02 on dist-test-slave-5n86"
server_key: "acd02ae665b197c9275d13e6e6488428"
server_key_iv: "e2e7d5a60b82b3ac1a10d9dd243a4409"
server_key_version: "encryptionkey@0"
I20260925 05:02:02.225250 9400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance:
uuid: "4a8bc636783f44d8adaa606389cf35f7"
format_stamp: "Formatted at 2026-09-25 05:02:02 on dist-test-slave-5n86"
server_key: "acd02ae665b197c9275d13e6e6488428"
server_key_iv: "e2e7d5a60b82b3ac1a10d9dd243a4409"
server_key_version: "encryptionkey@0"
I20260925 05:02:02.229842 9400 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260925 05:02:02.232991 9414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:02.234264 9400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 05:02:02.234426 9400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "4a8bc636783f44d8adaa606389cf35f7"
format_stamp: "Formatted at 2026-09-25 05:02:02 on dist-test-slave-5n86"
server_key: "acd02ae665b197c9275d13e6e6488428"
server_key_iv: "e2e7d5a60b82b3ac1a10d9dd243a4409"
server_key_version: "encryptionkey@0"
I20260925 05:02:02.234637 9400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:02:02.239722 9400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:02:02.240420 9400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:02:02.240707 9400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:02:02.241542 9400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:02:02.242781 9400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:02:02.242862 9400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:02.243006 9400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:02:02.243095 9400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:02.275035 9400 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:42535
I20260925 05:02:02.275130 9526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:42535 every 8 connection(s)
I20260925 05:02:02.276245 9400 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:02:02.284399 9119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:02.278741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:56051 (local address 127.30.247.62:39983)
0925 05:02:02.278928 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:02.278933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:02.280354 (+ 1421us) server_negotiation.cc:407] Connection header received
0925 05:02:02.281976 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:02.281982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:02.282049 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:02.282165 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:02.283688 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:02:02.283694 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:02:02.283700 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:02:02.283717 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:02:02.283760 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:02:02.283772 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:02:02.283904 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:02.284084 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 05:02:02.284148 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 05:02:02.285578 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 9400
I20260925 05:02:02.285773 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance
I20260925 05:02:02.286336 31708 external_mini_cluster.cc:1428] Setting key 86fa00cc4f9bbde30d7739cccc62ae02
I20260925 05:02:02.286440 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:02.279270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:39983 (local address 127.30.247.3:56051)
0925 05:02:02.280198 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0925 05:02:02.280233 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:02:02.281714 (+ 1481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:02:02.282378 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:02:02.282388 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:02:02.282650 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:02:02.283391 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:02:02.283394 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:02:02.283434 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:02:02.283513 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:02:02.283866 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:02:02.283876 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:02:02.283885 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 05:02:02.284161 (+ 276us) client_negotiation.cc:240] Negotiation successful
0925 05:02:02.284403 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":277,"threads_started":1}
I20260925 05:02:02.287875 9527 heartbeater.cc:347] Connected to a master server at 127.30.247.62:39983
I20260925 05:02:02.288153 9527 heartbeater.cc:464] Registering TS with master...
I20260925 05:02:02.288856 9527 heartbeater.cc:511] Master 127.30.247.62:39983 requested a full tablet report, sending...
I20260925 05:02:02.290230 9080 ts_manager.cc:194] Registered new tserver with Master: 4a8bc636783f44d8adaa606389cf35f7 (127.30.247.3:42535)
I20260925 05:02:02.301405 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 05:02:02.307318 9119 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:02:02.305764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44732 (local address 127.30.247.62:39983)
0925 05:02:02.305913 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:02.305917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:02.306404 (+ 487us) server_negotiation.cc:407] Connection header received
0925 05:02:02.306583 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:02.306593 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:02.306636 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:02.306734 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:02.307043 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44732: BlockingRecv error: recv got EOF from 127.0.0.1:44732 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 05:02:02.308135 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 9140
I20260925 05:02:02.311550 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:02.309100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48896 (local address 127.30.247.1:37963)
0925 05:02:02.309708 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:02.309720 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:02.309743 (+ 23us) server_negotiation.cc:407] Connection header received
0925 05:02:02.309794 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:02.309799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:02.309964 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:02.310077 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:02.310921 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:02:02.310927 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:02:02.310936 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:02:02.310952 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:02:02.310986 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:02:02.310999 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:02:02.311097 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:02.311175 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 05:02:02.311276 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":324,"threads_started":1}
I20260925 05:02:02.312687 9262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:02:02.394361 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 9140
I20260925 05:02:02.410902 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 9270
I20260925 05:02:02.414208 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:02.411730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55500 (local address 127.30.247.2:36341)
0925 05:02:02.412239 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:02.412246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:02.412272 (+ 26us) server_negotiation.cc:407] Connection header received
0925 05:02:02.412334 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:02.412340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:02.412611 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:02.412741 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:02.413612 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:02:02.413618 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:02:02.413628 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:02:02.413643 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:02:02.413682 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:02:02.413696 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:02:02.413805 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:02.413910 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 05:02:02.413988 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":252,"threads_started":1}
I20260925 05:02:02.415329 9392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:02:02.494059 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 9270
I20260925 05:02:02.509984 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 9400
I20260925 05:02:02.513895 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:02.510915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40782 (local address 127.30.247.3:42535)
0925 05:02:02.511474 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:02.511481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:02.511509 (+ 28us) server_negotiation.cc:407] Connection header received
0925 05:02:02.511574 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:02.511580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:02.511795 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:02.512062 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:02.513174 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:02:02.513180 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:02:02.513192 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:02:02.513203 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:02:02.513240 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:02:02.513256 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:02:02.513442 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:02.513510 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 05:02:02.513616 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":262,"threads_started":1}
I20260925 05:02:02.515087 9522 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:02:02.593986 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 9400
I20260925 05:02:02.609916 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 9050
I20260925 05:02:02.611039 9111 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:02:02.709096 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 9050
2026-09-25T05:02:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2238 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:02:02 dist-test-slave-5n86 krb5kdc[9545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:02:02 dist-test-slave-5n86 krb5kdc[9545](info): set up 2 sockets
Sep 25 05:02:02 dist-test-slave-5n86 krb5kdc[9545](info): commencing operation
krb5kdc: starting...
W20260925 05:02:03.797741 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:02:03 dist-test-slave-5n86 krb5kdc[9545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:02:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:02:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:02:04.012845 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:45647
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:45891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:45647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:02:04.317687 9558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:02:04.317936 9558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:02:04.318019 9558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:02:04.324910 9558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:02:04.325003 9558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:02:04.325047 9558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:02:04.325094 9558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:02:04.325138 9558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:02:04.334403 9558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:45891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:45647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:45647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:02:04.335530 9558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:02:04.336777 9558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:02:04.349262 9564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:02:04.348850 9563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:02:04.350026 9566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:02:04.350829 9558 server_base.cc:1034] running on GCE node
I20260925 05:02:04.351575 9558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:02:04.352998 9558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:02:04.354223 9558 hybrid_clock.cc:648] HybridClock initialized: now 1790312524354171 us; error 63 us; skew 500 ppm
I20260925 05:02:04.354575 9558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:02:04 dist-test-slave-5n86 krb5kdc[9545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312524, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:02:04.362592 9558 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:02:04.363971 9558 webserver.cc:460] Webserver started at http://127.30.247.62:34351/ using document root <none> and password file <none>
I20260925 05:02:04.364643 9558 fs_manager.cc:359] Metadata directory not provided
I20260925 05:02:04.364744 9558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:02:04.364981 9558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:02:04.367715 9558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "06a28207c8b043fdaef5fd0ed3288027"
format_stamp: "Formatted at 2026-09-25 05:02:04 on dist-test-slave-5n86"
server_key: "199636674abc575b70e49d636f6ce471"
server_key_iv: "6f1154efab00b9cd47242789e7894014"
server_key_version: "encryptionkey@0"
I20260925 05:02:04.368396 9558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "06a28207c8b043fdaef5fd0ed3288027"
format_stamp: "Formatted at 2026-09-25 05:02:04 on dist-test-slave-5n86"
server_key: "199636674abc575b70e49d636f6ce471"
server_key_iv: "6f1154efab00b9cd47242789e7894014"
server_key_version: "encryptionkey@0"
I20260925 05:02:04.373323 9558 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 05:02:04.376435 9573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:04.377724 9558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 05:02:04.377897 9558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "06a28207c8b043fdaef5fd0ed3288027"
format_stamp: "Formatted at 2026-09-25 05:02:04 on dist-test-slave-5n86"
server_key: "199636674abc575b70e49d636f6ce471"
server_key_iv: "6f1154efab00b9cd47242789e7894014"
server_key_version: "encryptionkey@0"
I20260925 05:02:04.378047 9558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:02:04.403137 9558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:02:04.406852 9558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:02:04.407114 9558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:02:04.425154 9558 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:45647
I20260925 05:02:04.425237 9624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:45647 every 8 connection(s)
I20260925 05:02:04.426450 9558 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:02:04.430089 9625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:02:04.430750 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 9558
I20260925 05:02:04.430997 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:02:04.431618 31708 external_mini_cluster.cc:1428] Setting key 33bc1c4d60967d715aceb7494546ce5b
I20260925 05:02:04.443723 9625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a28207c8b043fdaef5fd0ed3288027: Bootstrap starting.
Sep 25 05:02:04 dist-test-slave-5n86 krb5kdc[9545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:02:04.446645 9625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06a28207c8b043fdaef5fd0ed3288027: Neither blocks nor log segments found. Creating new log.
I20260925 05:02:04.447988 9625 log.cc:824] T 00000000000000000000000000000000 P 06a28207c8b043fdaef5fd0ed3288027: Log is configured to *not* fsync() on all Append() calls
I20260925 05:02:04.451293 9625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a28207c8b043fdaef5fd0ed3288027: No bootstrap required, opened a new log
I20260925 05:02:04.455442 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:04.433718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53040 (local address 127.30.247.62:45647)
0925 05:02:04.434481 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:04.434496 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:04.434544 (+ 48us) server_negotiation.cc:407] Connection header received
0925 05:02:04.436450 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:04.436479 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:04.436990 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:04.437424 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:04.438737 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:04.440727 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:04.441976 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:04.442961 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:04.443142 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:04.446954 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:02:04.446970 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:02:04.446992 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:02:04.447045 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:02:04.450997 (+ 3952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:04.451757 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:04.451765 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:04.451772 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:04.451969 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:04.452356 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:04.452362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:04.452366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:04.453025 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:02:04.453340 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:04.453729 (+ 389us) server_negotiation.cc:299] Negotiation successful
0925 05:02:04.454148 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":305,"threads_started":1}
I20260925 05:02:04.460275 9625 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06a28207c8b043fdaef5fd0ed3288027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a28207c8b043fdaef5fd0ed3288027" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45647 } }
I20260925 05:02:04.460629 9625 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06a28207c8b043fdaef5fd0ed3288027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:02:04.460740 9625 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06a28207c8b043fdaef5fd0ed3288027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06a28207c8b043fdaef5fd0ed3288027, State: Initialized, Role: FOLLOWER
I20260925 05:02:04.461336 9625 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06a28207c8b043fdaef5fd0ed3288027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a28207c8b043fdaef5fd0ed3288027" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45647 } }
I20260925 05:02:04.461613 9625 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06a28207c8b043fdaef5fd0ed3288027 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:02:04.461751 9625 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06a28207c8b043fdaef5fd0ed3288027 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:02:04.461925 9625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06a28207c8b043fdaef5fd0ed3288027 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:02:04.463865 9625 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06a28207c8b043fdaef5fd0ed3288027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a28207c8b043fdaef5fd0ed3288027" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45647 } }
I20260925 05:02:04.464313 9625 leader_election.cc:302] T 00000000000000000000000000000000 P 06a28207c8b043fdaef5fd0ed3288027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06a28207c8b043fdaef5fd0ed3288027; no voters:
I20260925 05:02:04.464968 9625 leader_election.cc:288] T 00000000000000000000000000000000 P 06a28207c8b043fdaef5fd0ed3288027 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:02:04.465121 9630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06a28207c8b043fdaef5fd0ed3288027 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:02:04.465787 9630 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06a28207c8b043fdaef5fd0ed3288027 [term 1 LEADER]: Becoming Leader. State: Replica: 06a28207c8b043fdaef5fd0ed3288027, State: Running, Role: LEADER
I20260925 05:02:04.466298 9630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06a28207c8b043fdaef5fd0ed3288027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a28207c8b043fdaef5fd0ed3288027" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45647 } }
I20260925 05:02:04.466894 9625 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06a28207c8b043fdaef5fd0ed3288027 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:02:04.469334 9631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a28207c8b043fdaef5fd0ed3288027 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06a28207c8b043fdaef5fd0ed3288027" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a28207c8b043fdaef5fd0ed3288027" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45647 } } }
I20260925 05:02:04.469676 9631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a28207c8b043fdaef5fd0ed3288027 [sys.catalog]: This master's current role is: LEADER
I20260925 05:02:04.470120 9632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a28207c8b043fdaef5fd0ed3288027 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06a28207c8b043fdaef5fd0ed3288027. Latest consensus state: current_term: 1 leader_uuid: "06a28207c8b043fdaef5fd0ed3288027" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a28207c8b043fdaef5fd0ed3288027" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45647 } } }
I20260925 05:02:04.470440 9632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a28207c8b043fdaef5fd0ed3288027 [sys.catalog]: This master's current role is: LEADER
I20260925 05:02:04.471276 9637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:02:04.476610 9637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:02:04.484576 9637 catalog_manager.cc:1347] Generated new cluster ID: 611f8f3fac3a4ceaa5946e09370d81c4
I20260925 05:02:04.484695 9637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:02:04.496110 9637 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:02:04.497675 9637 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:02:04.505888 9637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06a28207c8b043fdaef5fd0ed3288027: Generated new TSK 0
I20260925 05:02:04.506577 9637 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
I20260925 05:02:04.558310 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:0
--local_ip_for_outbound_sockets=127.30.247.1
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:45647
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:45891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:02:04.870649 9651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:02:04.870915 9651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:02:04.871069 9651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:02:04.878346 9651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:02:04.878443 9651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:02:04.878618 9651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:02:04.887982 9651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:45891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:45647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:02:04.889218 9651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:02:04.890631 9651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:02:04.903239 9656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:02:04.903285 9657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:02:04.903354 9659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:02:04.904510 9651 server_base.cc:1034] running on GCE node
I20260925 05:02:04.905123 9651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:02:04.906037 9651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:02:04.907265 9651 hybrid_clock.cc:648] HybridClock initialized: now 1790312524907193 us; error 64 us; skew 500 ppm
I20260925 05:02:04.907521 9651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:02:04 dist-test-slave-5n86 krb5kdc[9545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312524, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:02:04.910501 9651 init.cc:377] Logged in from keytab as kudu/127.30.247.1@KRBTEST.COM (short username kudu)
I20260925 05:02:04.911808 9651 webserver.cc:460] Webserver started at http://127.30.247.1:46265/ using document root <none> and password file <none>
I20260925 05:02:04.912446 9651 fs_manager.cc:359] Metadata directory not provided
I20260925 05:02:04.912626 9651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:02:04.912932 9651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:02:04.915603 9651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/data/instance:
uuid: "0d6b329f31654a509020ee35cac7b2b9"
format_stamp: "Formatted at 2026-09-25 05:02:04 on dist-test-slave-5n86"
server_key: "da470edde32aea1f3401ef77b8df51d8"
server_key_iv: "f8d3dca2e81627bad7428444515da532"
server_key_version: "encryptionkey@0"
I20260925 05:02:04.916338 9651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance:
uuid: "0d6b329f31654a509020ee35cac7b2b9"
format_stamp: "Formatted at 2026-09-25 05:02:04 on dist-test-slave-5n86"
server_key: "da470edde32aea1f3401ef77b8df51d8"
server_key_iv: "f8d3dca2e81627bad7428444515da532"
server_key_version: "encryptionkey@0"
I20260925 05:02:04.921056 9651 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 05:02:04.924176 9666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:04.925531 9651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:02:04.925685 9651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "0d6b329f31654a509020ee35cac7b2b9"
format_stamp: "Formatted at 2026-09-25 05:02:04 on dist-test-slave-5n86"
server_key: "da470edde32aea1f3401ef77b8df51d8"
server_key_iv: "f8d3dca2e81627bad7428444515da532"
server_key_version: "encryptionkey@0"
I20260925 05:02:04.925860 9651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:02:04.961673 9651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:02:04.965813 9651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:02:04.966104 9651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:02:04.967142 9651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:02:04.968398 9651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:02:04.968467 9651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:04.968657 9651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:02:04.968722 9651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:05.001642 9651 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:36717
I20260925 05:02:05.001688 9778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:36717 every 8 connection(s)
I20260925 05:02:05.002892 9651 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:02:05.007584 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 9651
I20260925 05:02:05.007835 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance
I20260925 05:02:05.008757 31708 external_mini_cluster.cc:1428] Setting key f06d24f7c900c0351e2bc55d92f57bf2
Sep 25 05:02:05 dist-test-slave-5n86 krb5kdc[9545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312524, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:02:05.025846 9781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:05.005378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:58697 (local address 127.30.247.62:45647)
0925 05:02:05.005967 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:05.005973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:05.006926 (+ 953us) server_negotiation.cc:407] Connection header received
0925 05:02:05.009650 (+ 2724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:05.009656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:05.009768 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:05.009894 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:05.011532 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:05.012237 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:05.014676 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:05.015119 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:05.015232 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:05.019174 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:02:05.019181 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:02:05.019185 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:02:05.019228 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:02:05.022713 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:05.023388 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:05.023395 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:05.023402 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:05.023538 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:05.023924 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:05.023932 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:05.023938 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:05.024295 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:02:05.024444 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:05.025114 (+ 670us) server_negotiation.cc:299] Negotiation successful
0925 05:02:05.025354 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":381,"threads_started":1}
I20260925 05:02:05.026325 9782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:05.006022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:45647 (local address 127.30.247.1:58697)
0925 05:02:05.006781 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0925 05:02:05.006821 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 05:02:05.009385 (+ 2564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:02:05.010118 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:02:05.010127 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:02:05.010594 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:02:05.011350 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:05.011374 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:05.012421 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:05.012429 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:05.014513 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:05.014530 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:05.015347 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:05.015356 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:05.015669 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:05.016368 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:02:05.016405 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:02:05.018959 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:02:05.022853 (+ 3894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:02:05.022859 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:02:05.022869 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:02:05.023260 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:02:05.023638 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:02:05.023641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:02:05.023643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:02:05.023813 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:02:05.024415 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:02:05.024421 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:02:05.024754 (+ 333us) client_negotiation.cc:769] Sending connection context
0925 05:02:05.024966 (+ 212us) client_negotiation.cc:240] Negotiation successful
0925 05:02:05.025305 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":245,"threads_started":1}
I20260925 05:02:05.027693 9779 heartbeater.cc:347] Connected to a master server at 127.30.247.62:45647
I20260925 05:02:05.027997 9779 heartbeater.cc:464] Registering TS with master...
I20260925 05:02:05.028861 9779 heartbeater.cc:511] Master 127.30.247.62:45647 requested a full tablet report, sending...
I20260925 05:02:05.031065 9590 ts_manager.cc:194] Registered new tserver with Master: 0d6b329f31654a509020ee35cac7b2b9 (127.30.247.1:36717)
I20260925 05:02:05.032426 9590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.1@KRBTEST.COM'} at 127.30.247.1:58697
WARNING: no policy specified for kudu/127.30.247.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
I20260925 05:02:05.059589 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:0
--local_ip_for_outbound_sockets=127.30.247.2
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:45647
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:45891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:02:05.374977 9785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:02:05.375216 9785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:02:05.375419 9785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:02:05.382632 9785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:02:05.382731 9785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:02:05.382936 9785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:02:05.392706 9785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:45891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:45647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:02:05.393826 9785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:02:05.395038 9785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:02:05.407893 9790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:02:05.407991 9791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:02:05.408401 9793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:02:05.408994 9785 server_base.cc:1034] running on GCE node
I20260925 05:02:05.409570 9785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:02:05.410475 9785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:02:05.411697 9785 hybrid_clock.cc:648] HybridClock initialized: now 1790312525411664 us; error 58 us; skew 500 ppm
I20260925 05:02:05.412004 9785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:02:05 dist-test-slave-5n86 krb5kdc[9545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312525, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:02:05.414961 9785 init.cc:377] Logged in from keytab as kudu/127.30.247.2@KRBTEST.COM (short username kudu)
I20260925 05:02:05.416326 9785 webserver.cc:460] Webserver started at http://127.30.247.2:40157/ using document root <none> and password file <none>
I20260925 05:02:05.417024 9785 fs_manager.cc:359] Metadata directory not provided
I20260925 05:02:05.417166 9785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:02:05.417456 9785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:02:05.420125 9785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/data/instance:
uuid: "a28aec76a3204f7daf88d19d7781d18f"
format_stamp: "Formatted at 2026-09-25 05:02:05 on dist-test-slave-5n86"
server_key: "bf7b975c578beaab21f63e2d26e097f3"
server_key_iv: "83a7e4368a9664ea6260e989f6688946"
server_key_version: "encryptionkey@0"
I20260925 05:02:05.420918 9785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance:
uuid: "a28aec76a3204f7daf88d19d7781d18f"
format_stamp: "Formatted at 2026-09-25 05:02:05 on dist-test-slave-5n86"
server_key: "bf7b975c578beaab21f63e2d26e097f3"
server_key_iv: "83a7e4368a9664ea6260e989f6688946"
server_key_version: "encryptionkey@0"
I20260925 05:02:05.425807 9785 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 05:02:05.429044 9800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:05.430585 9785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 05:02:05.430744 9785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "a28aec76a3204f7daf88d19d7781d18f"
format_stamp: "Formatted at 2026-09-25 05:02:05 on dist-test-slave-5n86"
server_key: "bf7b975c578beaab21f63e2d26e097f3"
server_key_iv: "83a7e4368a9664ea6260e989f6688946"
server_key_version: "encryptionkey@0"
I20260925 05:02:05.430940 9785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:02:05.462637 9785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:02:05.466223 9785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:02:05.466508 9785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:02:05.467418 9785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:02:05.468688 9785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:02:05.468765 9785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:05.468844 9785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:02:05.468894 9785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:05.499519 9785 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:44075
I20260925 05:02:05.499599 9912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:44075 every 8 connection(s)
I20260925 05:02:05.500761 9785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:02:05.509130 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 9785
I20260925 05:02:05.509339 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance
I20260925 05:02:05.509754 31708 external_mini_cluster.cc:1428] Setting key 9551bd767da1c0810bdc14070ccabdd9
Sep 25 05:02:05 dist-test-slave-5n86 krb5kdc[9545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312525, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:02:05.520409 9781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:05.503149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:41237 (local address 127.30.247.62:45647)
0925 05:02:05.503334 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:05.503339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:05.504392 (+ 1053us) server_negotiation.cc:407] Connection header received
0925 05:02:05.506207 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:05.506212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:05.506306 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:05.506418 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:05.507764 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:05.508452 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:05.510126 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:05.510707 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:05.510823 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:05.514911 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:02:05.514919 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:02:05.514924 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:02:05.514964 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:02:05.517586 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:05.518289 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:05.518294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:05.518298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:05.518471 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:05.518904 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:05.518911 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:05.518915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:05.519289 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:02:05.519445 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:05.520014 (+ 569us) server_negotiation.cc:299] Negotiation successful
0925 05:02:05.520170 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 05:02:05.525353 9915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:05.503600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:45647 (local address 127.30.247.2:41237)
0925 05:02:05.504245 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0925 05:02:05.504281 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 05:02:05.505945 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:02:05.506640 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:02:05.506647 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:02:05.507102 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:02:05.507605 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:05.507621 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:05.508625 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:05.508629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:05.509986 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:05.509997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:05.510914 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:05.510918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:05.511107 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:05.511841 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:02:05.511868 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:02:05.514628 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:02:05.517752 (+ 3124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:02:05.517760 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:02:05.517774 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:02:05.518156 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:02:05.518596 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:02:05.518600 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:02:05.518602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:02:05.518783 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:02:05.519416 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:02:05.519422 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:02:05.519693 (+ 271us) client_negotiation.cc:769] Sending connection context
0925 05:02:05.519910 (+ 217us) client_negotiation.cc:240] Negotiation successful
0925 05:02:05.520267 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":280,"threads_started":1}
I20260925 05:02:05.526814 9913 heartbeater.cc:347] Connected to a master server at 127.30.247.62:45647
I20260925 05:02:05.527103 9913 heartbeater.cc:464] Registering TS with master...
I20260925 05:02:05.527822 9913 heartbeater.cc:511] Master 127.30.247.62:45647 requested a full tablet report, sending...
I20260925 05:02:05.529407 9590 ts_manager.cc:194] Registered new tserver with Master: a28aec76a3204f7daf88d19d7781d18f (127.30.247.2:44075)
I20260925 05:02:05.530658 9590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.2@KRBTEST.COM'} at 127.30.247.2:41237
WARNING: no policy specified for kudu/127.30.247.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
I20260925 05:02:05.559381 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:0
--local_ip_for_outbound_sockets=127.30.247.3
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:45647
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:45891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:02:05.866209 9918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:02:05.866536 9918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:02:05.866731 9918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:02:05.874100 9918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:02:05.874195 9918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:02:05.874402 9918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:02:05.883982 9918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:45891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:45647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:02:05.885210 9918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:02:05.886439 9918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:02:05.898919 9924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:02:05.898928 9926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:02:05.898968 9923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:02:05.900193 9918 server_base.cc:1034] running on GCE node
I20260925 05:02:05.900823 9918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:02:05.901664 9918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:02:05.902870 9918 hybrid_clock.cc:648] HybridClock initialized: now 1790312525902832 us; error 60 us; skew 500 ppm
I20260925 05:02:05.903155 9918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:02:05 dist-test-slave-5n86 krb5kdc[9545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312525, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:02:05.906060 9918 init.cc:377] Logged in from keytab as kudu/127.30.247.3@KRBTEST.COM (short username kudu)
I20260925 05:02:05.907399 9918 webserver.cc:460] Webserver started at http://127.30.247.3:46649/ using document root <none> and password file <none>
I20260925 05:02:05.908041 9918 fs_manager.cc:359] Metadata directory not provided
I20260925 05:02:05.908176 9918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:02:05.908456 9918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:02:05.911134 9918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/data/instance:
uuid: "71c4f6c339554dd797ee08a4e553ea09"
format_stamp: "Formatted at 2026-09-25 05:02:05 on dist-test-slave-5n86"
server_key: "cda6bfbb20952c2368578830bd3fa652"
server_key_iv: "56f00efd30ff6f07a7fa294f5d405296"
server_key_version: "encryptionkey@0"
I20260925 05:02:05.911963 9918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance:
uuid: "71c4f6c339554dd797ee08a4e553ea09"
format_stamp: "Formatted at 2026-09-25 05:02:05 on dist-test-slave-5n86"
server_key: "cda6bfbb20952c2368578830bd3fa652"
server_key_iv: "56f00efd30ff6f07a7fa294f5d405296"
server_key_version: "encryptionkey@0"
I20260925 05:02:05.916712 9918 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260925 05:02:05.919832 9933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:05.921305 9918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:02:05.921475 9918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "71c4f6c339554dd797ee08a4e553ea09"
format_stamp: "Formatted at 2026-09-25 05:02:05 on dist-test-slave-5n86"
server_key: "cda6bfbb20952c2368578830bd3fa652"
server_key_iv: "56f00efd30ff6f07a7fa294f5d405296"
server_key_version: "encryptionkey@0"
I20260925 05:02:05.921665 9918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:02:05.933597 9918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:02:05.937077 9918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:02:05.937354 9918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:02:05.938171 9918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:02:05.939374 9918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:02:05.939476 9918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:05.939622 9918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:02:05.939693 9918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:05.974620 9918 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:46605
I20260925 05:02:05.974705 10045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:46605 every 8 connection(s)
I20260925 05:02:05.975809 9918 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:02:05.977319 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 9918
I20260925 05:02:05.977533 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance
I20260925 05:02:05.978037 31708 external_mini_cluster.cc:1428] Setting key e78c95910abf0609427da21a97158c78
Sep 25 05:02:05 dist-test-slave-5n86 krb5kdc[9545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312525, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:02:05.995639 9781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:05.978640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:55829 (local address 127.30.247.62:45647)
0925 05:02:05.978828 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:05.978832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:05.980247 (+ 1415us) server_negotiation.cc:407] Connection header received
0925 05:02:05.982257 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:05.982261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:05.982374 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:05.982465 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:05.983832 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:05.984646 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:05.986125 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:05.986583 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:05.986684 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:05.990103 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:02:05.990108 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:02:05.990111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:02:05.990141 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:02:05.993013 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:05.993746 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:05.993751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:05.993754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:05.993879 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:05.994269 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:05.994273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:05.994275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:05.994537 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:02:05.994644 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:05.995236 (+ 592us) server_negotiation.cc:299] Negotiation successful
0925 05:02:05.995404 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 05:02:05.996618 10048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:05.979252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:45647 (local address 127.30.247.3:55829)
0925 05:02:05.980078 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0925 05:02:05.980129 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 05:02:05.982034 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:02:05.982679 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:02:05.982687 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:02:05.983150 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:02:05.983659 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:05.983675 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:05.984803 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:05.984807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:05.985996 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:05.986012 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:05.986801 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:05.986806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:05.986997 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:05.987685 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:02:05.987707 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:02:05.989903 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:02:05.993185 (+ 3282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:02:05.993197 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:02:05.993214 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:02:05.993640 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:02:05.993977 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:02:05.993988 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:02:05.993990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:02:05.994153 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:02:05.994637 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:02:05.994643 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:02:05.994932 (+ 289us) client_negotiation.cc:769] Sending connection context
0925 05:02:05.995156 (+ 224us) client_negotiation.cc:240] Negotiation successful
0925 05:02:05.995501 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":323,"threads_started":1}
I20260925 05:02:05.997891 10046 heartbeater.cc:347] Connected to a master server at 127.30.247.62:45647
I20260925 05:02:05.998137 10046 heartbeater.cc:464] Registering TS with master...
I20260925 05:02:05.998804 10046 heartbeater.cc:511] Master 127.30.247.62:45647 requested a full tablet report, sending...
I20260925 05:02:06.000087 9590 ts_manager.cc:194] Registered new tserver with Master: 71c4f6c339554dd797ee08a4e553ea09 (127.30.247.3:46605)
I20260925 05:02:06.000923 9590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.3@KRBTEST.COM'} at 127.30.247.3:55829
I20260925 05:02:06.005622 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:02:06.021564 9781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:06.009812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53052 (local address 127.30.247.62:45647)
0925 05:02:06.009957 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:06.009961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:06.010461 (+ 500us) server_negotiation.cc:407] Connection header received
0925 05:02:06.010568 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:06.010571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:06.010624 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:06.010698 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:06.011957 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.012763 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.013943 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.014345 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.014457 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:06.016772 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:02:06.016776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:02:06.016779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:02:06.016808 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:02:06.019293 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:06.019906 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:06.019910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:06.019912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:06.020042 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:06.020390 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:06.020393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:06.020395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:06.020768 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:02:06.020940 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:06.021214 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 05:02:06.021358 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 05:02:06.025285 9590 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:02:06.032577 9590 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:02:06.035038 9779 heartbeater.cc:503] Master 127.30.247.62:45647 was elected leader, sending a full tablet report...
I20260925 05:02:06.068097 10060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:06.044082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:46605 (local address 127.0.0.1:41596)
0925 05:02:06.053860 (+ 9778us) negotiation.cc:107] Waiting for socket to connect
0925 05:02:06.053906 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 05:02:06.054059 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:02:06.060713 (+ 6654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:02:06.060719 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:02:06.060747 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:02:06.061262 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:06.061281 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.062614 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.062622 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:06.065065 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:06.065079 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.067094 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.067101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:06.067323 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:06.067371 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 05:02:06.067489 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 05:02:06.067640 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9538,"mutex_wait_us":31,"thread_start_us":15726,"threads_started":1}
I20260925 05:02:06.068317 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:06.045218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60638 (local address 127.30.247.2:44075)
0925 05:02:06.052721 (+ 7503us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:06.052743 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:06.060276 (+ 7533us) server_negotiation.cc:407] Connection header received
0925 05:02:06.060380 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:06.060386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:06.060604 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:06.060785 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:02:06.061793 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.063555 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.065771 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.067163 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.067300 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:06.067468 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:06.067806 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 05:02:06.067955 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7295,"thread_start_us":7130,"threads_started":1}
I20260925 05:02:06.068935 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:06.044335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.30.247.3:46605)
0925 05:02:06.059809 (+ 15474us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:06.059819 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:06.059856 (+ 37us) server_negotiation.cc:407] Connection header received
0925 05:02:06.059967 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:06.059975 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:06.060225 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:06.060421 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:02:06.061446 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.062481 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.065227 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.066624 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.066748 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:06.066971 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:06.068581 (+ 1610us) server_negotiation.cc:299] Negotiation successful
0925 05:02:06.068712 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15311,"thread_start_us":15162,"threads_started":1}
I20260925 05:02:06.069165 10062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:06.045033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32824 (local address 127.30.247.1:36717)
0925 05:02:06.059883 (+ 14850us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:06.059891 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:06.059918 (+ 27us) server_negotiation.cc:407] Connection header received
0925 05:02:06.060022 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:06.060031 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:06.060266 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:06.061756 (+ 1490us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:02:06.061987 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.063493 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.065891 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.067333 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.068558 (+ 1225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:06.068734 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:06.068786 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 05:02:06.068889 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14658,"thread_start_us":15368,"threads_started":1}
I20260925 05:02:06.068431 10058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:06.044663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:44075 (local address 127.0.0.1:60638)
0925 05:02:06.060043 (+ 15380us) negotiation.cc:107] Waiting for socket to connect
0925 05:02:06.060059 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:02:06.060164 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:02:06.060766 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:02:06.060772 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:02:06.060798 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:02:06.061588 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:06.061597 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.063721 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.063726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:06.065572 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:06.065593 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.067408 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.067416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:06.067598 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:06.067632 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 05:02:06.067719 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 05:02:06.067880 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15278,"mutex_wait_us":133,"thread_start_us":14462,"threads_started":1}
I20260925 05:02:06.068454 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:06.044402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:36717 (local address 127.0.0.1:32824)
0925 05:02:06.058870 (+ 14468us) negotiation.cc:107] Waiting for socket to connect
0925 05:02:06.058893 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 05:02:06.059035 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:02:06.060673 (+ 1638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:02:06.060682 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:02:06.060715 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:02:06.061573 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:06.061587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.063661 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.063668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:06.065744 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:06.065755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.067554 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.067561 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:06.067706 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:06.067736 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 05:02:06.067793 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 05:02:06.067917 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14288,"mutex_wait_us":149,"thread_start_us":9075,"threads_started":1}
I20260925 05:02:06.072791 9714 tablet_service.cc:1467] Processing CreateTablet for tablet c7dafb9e55214818913a6f1b37a96d99 (DEFAULT_TABLE table=test-table [id=165c52b7fc2d46fd9ce2dabd6697290c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:02:06.072805 9848 tablet_service.cc:1467] Processing CreateTablet for tablet c7dafb9e55214818913a6f1b37a96d99 (DEFAULT_TABLE table=test-table [id=165c52b7fc2d46fd9ce2dabd6697290c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:02:06.072804 9981 tablet_service.cc:1467] Processing CreateTablet for tablet c7dafb9e55214818913a6f1b37a96d99 (DEFAULT_TABLE table=test-table [id=165c52b7fc2d46fd9ce2dabd6697290c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:02:06.073958 9714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7dafb9e55214818913a6f1b37a96d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:02:06.074081 9981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7dafb9e55214818913a6f1b37a96d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:02:06.074082 9848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7dafb9e55214818913a6f1b37a96d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:02:06.084910 10064 tablet_bootstrap.cc:492] T c7dafb9e55214818913a6f1b37a96d99 P 0d6b329f31654a509020ee35cac7b2b9: Bootstrap starting.
I20260925 05:02:06.087944 10066 tablet_bootstrap.cc:492] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09: Bootstrap starting.
I20260925 05:02:06.088083 10064 tablet_bootstrap.cc:654] T c7dafb9e55214818913a6f1b37a96d99 P 0d6b329f31654a509020ee35cac7b2b9: Neither blocks nor log segments found. Creating new log.
I20260925 05:02:06.089057 10065 tablet_bootstrap.cc:492] T c7dafb9e55214818913a6f1b37a96d99 P a28aec76a3204f7daf88d19d7781d18f: Bootstrap starting.
I20260925 05:02:06.089763 10064 log.cc:824] T c7dafb9e55214818913a6f1b37a96d99 P 0d6b329f31654a509020ee35cac7b2b9: Log is configured to *not* fsync() on all Append() calls
I20260925 05:02:06.092252 10066 tablet_bootstrap.cc:654] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09: Neither blocks nor log segments found. Creating new log.
I20260925 05:02:06.092288 10065 tablet_bootstrap.cc:654] T c7dafb9e55214818913a6f1b37a96d99 P a28aec76a3204f7daf88d19d7781d18f: Neither blocks nor log segments found. Creating new log.
I20260925 05:02:06.092866 10064 tablet_bootstrap.cc:492] T c7dafb9e55214818913a6f1b37a96d99 P 0d6b329f31654a509020ee35cac7b2b9: No bootstrap required, opened a new log
I20260925 05:02:06.093113 10064 ts_tablet_manager.cc:1397] T c7dafb9e55214818913a6f1b37a96d99 P 0d6b329f31654a509020ee35cac7b2b9: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 05:02:06.093410 10066 log.cc:824] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09: Log is configured to *not* fsync() on all Append() calls
I20260925 05:02:06.093485 10065 log.cc:824] T c7dafb9e55214818913a6f1b37a96d99 P a28aec76a3204f7daf88d19d7781d18f: Log is configured to *not* fsync() on all Append() calls
I20260925 05:02:06.095945 10065 tablet_bootstrap.cc:492] T c7dafb9e55214818913a6f1b37a96d99 P a28aec76a3204f7daf88d19d7781d18f: No bootstrap required, opened a new log
I20260925 05:02:06.096194 10066 tablet_bootstrap.cc:492] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09: No bootstrap required, opened a new log
I20260925 05:02:06.096190 10065 ts_tablet_manager.cc:1397] T c7dafb9e55214818913a6f1b37a96d99 P a28aec76a3204f7daf88d19d7781d18f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260925 05:02:06.096427 10066 ts_tablet_manager.cc:1397] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260925 05:02:06.103307 10065 raft_consensus.cc:348] T c7dafb9e55214818913a6f1b37a96d99 P a28aec76a3204f7daf88d19d7781d18f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c4f6c339554dd797ee08a4e553ea09" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 46605 } } peers { permanent_uuid: "0d6b329f31654a509020ee35cac7b2b9" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36717 } } peers { permanent_uuid: "a28aec76a3204f7daf88d19d7781d18f" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44075 } }
I20260925 05:02:06.103647 10065 raft_consensus.cc:374] T c7dafb9e55214818913a6f1b37a96d99 P a28aec76a3204f7daf88d19d7781d18f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:02:06.103421 10064 raft_consensus.cc:348] T c7dafb9e55214818913a6f1b37a96d99 P 0d6b329f31654a509020ee35cac7b2b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c4f6c339554dd797ee08a4e553ea09" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 46605 } } peers { permanent_uuid: "0d6b329f31654a509020ee35cac7b2b9" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36717 } } peers { permanent_uuid: "a28aec76a3204f7daf88d19d7781d18f" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44075 } }
I20260925 05:02:06.103783 10065 raft_consensus.cc:729] T c7dafb9e55214818913a6f1b37a96d99 P a28aec76a3204f7daf88d19d7781d18f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a28aec76a3204f7daf88d19d7781d18f, State: Initialized, Role: FOLLOWER
I20260925 05:02:06.103806 10064 raft_consensus.cc:374] T c7dafb9e55214818913a6f1b37a96d99 P 0d6b329f31654a509020ee35cac7b2b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:02:06.103888 10064 raft_consensus.cc:729] T c7dafb9e55214818913a6f1b37a96d99 P 0d6b329f31654a509020ee35cac7b2b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d6b329f31654a509020ee35cac7b2b9, State: Initialized, Role: FOLLOWER
I20260925 05:02:06.104374 10065 consensus_queue.cc:260] T c7dafb9e55214818913a6f1b37a96d99 P a28aec76a3204f7daf88d19d7781d18f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c4f6c339554dd797ee08a4e553ea09" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 46605 } } peers { permanent_uuid: "0d6b329f31654a509020ee35cac7b2b9" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36717 } } peers { permanent_uuid: "a28aec76a3204f7daf88d19d7781d18f" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44075 } }
I20260925 05:02:06.104521 10064 consensus_queue.cc:260] T c7dafb9e55214818913a6f1b37a96d99 P 0d6b329f31654a509020ee35cac7b2b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c4f6c339554dd797ee08a4e553ea09" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 46605 } } peers { permanent_uuid: "0d6b329f31654a509020ee35cac7b2b9" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36717 } } peers { permanent_uuid: "a28aec76a3204f7daf88d19d7781d18f" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44075 } }
I20260925 05:02:06.105461 9913 heartbeater.cc:503] Master 127.30.247.62:45647 was elected leader, sending a full tablet report...
I20260925 05:02:06.105918 10065 ts_tablet_manager.cc:1428] T c7dafb9e55214818913a6f1b37a96d99 P a28aec76a3204f7daf88d19d7781d18f: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260925 05:02:06.108978 10066 raft_consensus.cc:348] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c4f6c339554dd797ee08a4e553ea09" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 46605 } } peers { permanent_uuid: "0d6b329f31654a509020ee35cac7b2b9" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36717 } } peers { permanent_uuid: "a28aec76a3204f7daf88d19d7781d18f" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44075 } }
I20260925 05:02:06.109525 10066 raft_consensus.cc:374] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:02:06.109648 10066 raft_consensus.cc:729] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c4f6c339554dd797ee08a4e553ea09, State: Initialized, Role: FOLLOWER
I20260925 05:02:06.110335 10066 consensus_queue.cc:260] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c4f6c339554dd797ee08a4e553ea09" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 46605 } } peers { permanent_uuid: "0d6b329f31654a509020ee35cac7b2b9" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36717 } } peers { permanent_uuid: "a28aec76a3204f7daf88d19d7781d18f" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44075 } }
I20260925 05:02:06.111487 10046 heartbeater.cc:503] Master 127.30.247.62:45647 was elected leader, sending a full tablet report...
I20260925 05:02:06.112094 10066 ts_tablet_manager.cc:1428] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
I20260925 05:02:06.112491 10064 ts_tablet_manager.cc:1428] T c7dafb9e55214818913a6f1b37a96d99 P 0d6b329f31654a509020ee35cac7b2b9: Time spent starting tablet: real 0.019s user 0.014s sys 0.004s
I20260925 05:02:06.190764 10072 raft_consensus.cc:482] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:02:06.191061 10072 raft_consensus.cc:504] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c4f6c339554dd797ee08a4e553ea09" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 46605 } } peers { permanent_uuid: "0d6b329f31654a509020ee35cac7b2b9" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36717 } } peers { permanent_uuid: "a28aec76a3204f7daf88d19d7781d18f" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44075 } }
I20260925 05:02:06.192777 10072 leader_election.cc:288] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d6b329f31654a509020ee35cac7b2b9 (127.30.247.1:36717), a28aec76a3204f7daf88d19d7781d18f (127.30.247.2:44075)
I20260925 05:02:06.199031 10048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:06.193175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:36717 (local address 127.30.247.3:55921)
0925 05:02:06.193374 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 05:02:06.193397 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 05:02:06.193547 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:02:06.193889 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:02:06.193894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:02:06.193920 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:02:06.194122 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:06.194131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.195356 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.195360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:06.197291 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:06.197306 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.198510 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.198513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:06.198645 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:06.198669 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 05:02:06.198751 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 05:02:06.198835 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":85}
I20260925 05:02:06.199156 10062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:06.193204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:55921 (local address 127.30.247.1:36717)
0925 05:02:06.193400 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:06.193406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:06.193487 (+ 81us) server_negotiation.cc:407] Connection header received
0925 05:02:06.193693 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:06.193698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:06.193768 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:06.193867 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:02:06.194262 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.195221 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.197436 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.198370 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.198520 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:06.198585 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:06.198790 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 05:02:06.198920 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 05:02:06.200114 10073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:06.193195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:44075 (local address 127.30.247.3:50089)
0925 05:02:06.194213 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0925 05:02:06.194232 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:02:06.194330 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:02:06.194636 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:02:06.194639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:02:06.194656 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:02:06.194937 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:06.194944 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.196010 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.196015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:06.197858 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:06.197869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.199480 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.199486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:06.199653 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:06.199691 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 05:02:06.199786 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 05:02:06.199864 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":915,"mutex_wait_us":261,"thread_start_us":439,"threads_started":1}
I20260925 05:02:06.200255 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:06.193392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:50089 (local address 127.30.247.2:44075)
0925 05:02:06.193594 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:06.193601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:06.194299 (+ 698us) server_negotiation.cc:407] Connection header received
0925 05:02:06.194445 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:06.194450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:06.194526 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:06.194613 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:02:06.195094 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.195851 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.198011 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.199175 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.199326 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:06.199391 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:06.199869 (+ 478us) server_negotiation.cc:299] Negotiation successful
0925 05:02:06.200008 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 05:02:06.200223 9734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7dafb9e55214818913a6f1b37a96d99" candidate_uuid: "71c4f6c339554dd797ee08a4e553ea09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d6b329f31654a509020ee35cac7b2b9" is_pre_election: true
I20260925 05:02:06.200843 9734 raft_consensus.cc:2454] T c7dafb9e55214818913a6f1b37a96d99 P 0d6b329f31654a509020ee35cac7b2b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71c4f6c339554dd797ee08a4e553ea09 in term 0.
I20260925 05:02:06.201181 9868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7dafb9e55214818913a6f1b37a96d99" candidate_uuid: "71c4f6c339554dd797ee08a4e553ea09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a28aec76a3204f7daf88d19d7781d18f" is_pre_election: true
I20260925 05:02:06.201635 9936 leader_election.cc:302] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d6b329f31654a509020ee35cac7b2b9, 71c4f6c339554dd797ee08a4e553ea09; no voters:
I20260925 05:02:06.201670 9868 raft_consensus.cc:2454] T c7dafb9e55214818913a6f1b37a96d99 P a28aec76a3204f7daf88d19d7781d18f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71c4f6c339554dd797ee08a4e553ea09 in term 0.
I20260925 05:02:06.202028 10072 raft_consensus.cc:2781] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:02:06.202191 10072 raft_consensus.cc:482] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:02:06.202335 10072 raft_consensus.cc:3037] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:02:06.204416 10072 raft_consensus.cc:504] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c4f6c339554dd797ee08a4e553ea09" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 46605 } } peers { permanent_uuid: "0d6b329f31654a509020ee35cac7b2b9" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36717 } } peers { permanent_uuid: "a28aec76a3204f7daf88d19d7781d18f" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44075 } }
I20260925 05:02:06.204998 10072 leader_election.cc:288] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09 [CANDIDATE]: Term 1 election: Requested vote from peers 0d6b329f31654a509020ee35cac7b2b9 (127.30.247.1:36717), a28aec76a3204f7daf88d19d7781d18f (127.30.247.2:44075)
I20260925 05:02:06.205323 9734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7dafb9e55214818913a6f1b37a96d99" candidate_uuid: "71c4f6c339554dd797ee08a4e553ea09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d6b329f31654a509020ee35cac7b2b9"
I20260925 05:02:06.205430 9868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7dafb9e55214818913a6f1b37a96d99" candidate_uuid: "71c4f6c339554dd797ee08a4e553ea09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a28aec76a3204f7daf88d19d7781d18f"
I20260925 05:02:06.205504 9734 raft_consensus.cc:3037] T c7dafb9e55214818913a6f1b37a96d99 P 0d6b329f31654a509020ee35cac7b2b9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:02:06.205646 9868 raft_consensus.cc:3037] T c7dafb9e55214818913a6f1b37a96d99 P a28aec76a3204f7daf88d19d7781d18f [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:02:06.207736 9734 raft_consensus.cc:2454] T c7dafb9e55214818913a6f1b37a96d99 P 0d6b329f31654a509020ee35cac7b2b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71c4f6c339554dd797ee08a4e553ea09 in term 1.
I20260925 05:02:06.207971 9868 raft_consensus.cc:2454] T c7dafb9e55214818913a6f1b37a96d99 P a28aec76a3204f7daf88d19d7781d18f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71c4f6c339554dd797ee08a4e553ea09 in term 1.
I20260925 05:02:06.208137 9936 leader_election.cc:302] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d6b329f31654a509020ee35cac7b2b9, 71c4f6c339554dd797ee08a4e553ea09; no voters:
I20260925 05:02:06.208395 10072 raft_consensus.cc:2781] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:02:06.208866 10072 raft_consensus.cc:686] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09 [term 1 LEADER]: Becoming Leader. State: Replica: 71c4f6c339554dd797ee08a4e553ea09, State: Running, Role: LEADER
I20260925 05:02:06.209414 10072 consensus_queue.cc:237] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c4f6c339554dd797ee08a4e553ea09" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 46605 } } peers { permanent_uuid: "0d6b329f31654a509020ee35cac7b2b9" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36717 } } peers { permanent_uuid: "a28aec76a3204f7daf88d19d7781d18f" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44075 } }
I20260925 05:02:06.213969 9590 catalog_manager.cc:5495] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71c4f6c339554dd797ee08a4e553ea09 (127.30.247.3). New cstate: current_term: 1 leader_uuid: "71c4f6c339554dd797ee08a4e553ea09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c4f6c339554dd797ee08a4e553ea09" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d6b329f31654a509020ee35cac7b2b9" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a28aec76a3204f7daf88d19d7781d18f" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44075 } health_report { overall_health: UNKNOWN } } }
W20260925 05:02:06.227293 10047 tablet.cc:2369] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:02:06.252774 9914 tablet.cc:2369] T c7dafb9e55214818913a6f1b37a96d99 P a28aec76a3204f7daf88d19d7781d18f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:02:06.255175 9780 tablet.cc:2369] T c7dafb9e55214818913a6f1b37a96d99 P 0d6b329f31654a509020ee35cac7b2b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:02:06.258322 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:06.252650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41608 (local address 127.30.247.3:46605)
0925 05:02:06.252841 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:06.252846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:06.253143 (+ 297us) server_negotiation.cc:407] Connection header received
0925 05:02:06.253400 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:06.253404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:06.253481 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:06.253640 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:02:06.253976 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.254705 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.256093 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.256485 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.256661 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:06.257215 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:02:06.257727 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:02:06.257846 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:06.257970 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 05:02:06.258036 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 05:02:06.266089 9868 raft_consensus.cc:1261] T c7dafb9e55214818913a6f1b37a96d99 P a28aec76a3204f7daf88d19d7781d18f [term 1 FOLLOWER]: Refusing update from remote peer 71c4f6c339554dd797ee08a4e553ea09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:02:06.266103 9734 raft_consensus.cc:1261] T c7dafb9e55214818913a6f1b37a96d99 P 0d6b329f31654a509020ee35cac7b2b9 [term 1 FOLLOWER]: Refusing update from remote peer 71c4f6c339554dd797ee08a4e553ea09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:02:06.267112 10074 consensus_queue.cc:1035] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a28aec76a3204f7daf88d19d7781d18f" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:02:06.267437 10072 consensus_queue.cc:1035] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d6b329f31654a509020ee35cac7b2b9" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:02:06.284569 10078 mvcc.cc:205] Tried to move back new op lower bound from 7333120107576336384 to 7333120107358048256. Current Snapshot: MvccSnapshot[applied={T|T < 7333120107576336384 or (T in {7333120107576336384})}]
I20260925 05:02:06.293079 9590 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:02:06.293321 9590 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:02:06.296589 9590 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06a28207c8b043fdaef5fd0ed3288027: Sending DeleteTablet for 3 replicas of tablet c7dafb9e55214818913a6f1b37a96d99
I20260925 05:02:06.297621 9981 tablet_service.cc:1514] Processing DeleteTablet for tablet c7dafb9e55214818913a6f1b37a96d99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:02:06 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:41596
I20260925 05:02:06.297806 9848 tablet_service.cc:1514] Processing DeleteTablet for tablet c7dafb9e55214818913a6f1b37a96d99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:02:06 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:60638
I20260925 05:02:06.298256 9714 tablet_service.cc:1514] Processing DeleteTablet for tablet c7dafb9e55214818913a6f1b37a96d99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:02:06 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:32824
I20260925 05:02:06.299346 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 9651
I20260925 05:02:06.310550 10086 tablet_replica.cc:330] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09: stopping tablet replica
I20260925 05:02:06.311072 10087 tablet_replica.cc:330] T c7dafb9e55214818913a6f1b37a96d99 P a28aec76a3204f7daf88d19d7781d18f: stopping tablet replica
I20260925 05:02:06.311578 10087 raft_consensus.cc:2229] T c7dafb9e55214818913a6f1b37a96d99 P a28aec76a3204f7daf88d19d7781d18f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:02:06.312081 10086 raft_consensus.cc:2229] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:02:06.312630 10087 raft_consensus.cc:2258] T c7dafb9e55214818913a6f1b37a96d99 P a28aec76a3204f7daf88d19d7781d18f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:02:06.313230 10086 raft_consensus.cc:2258] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:02:06.315506 10086 ts_tablet_manager.cc:1905] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:02:06.316170 10087 ts_tablet_manager.cc:1905] T c7dafb9e55214818913a6f1b37a96d99 P a28aec76a3204f7daf88d19d7781d18f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:02:06.322538 10086 ts_tablet_manager.cc:1918] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:02:06.322765 10086 log.cc:1196] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-2/wal/wals/c7dafb9e55214818913a6f1b37a96d99
I20260925 05:02:06.323318 10086 ts_tablet_manager.cc:1939] T c7dafb9e55214818913a6f1b37a96d99 P 71c4f6c339554dd797ee08a4e553ea09: Deleting consensus metadata
I20260925 05:02:06.325310 10088 tablet_replica.cc:330] T c7dafb9e55214818913a6f1b37a96d99 P 0d6b329f31654a509020ee35cac7b2b9: stopping tablet replica
I20260925 05:02:06.325780 10088 raft_consensus.cc:2229] T c7dafb9e55214818913a6f1b37a96d99 P 0d6b329f31654a509020ee35cac7b2b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:02:06.325838 9577 catalog_manager.cc:4841] TS 71c4f6c339554dd797ee08a4e553ea09 (127.30.247.3:46605): tablet c7dafb9e55214818913a6f1b37a96d99 (table test-table [id=165c52b7fc2d46fd9ce2dabd6697290c]) successfully deleted
I20260925 05:02:06.326280 10088 raft_consensus.cc:2258] T c7dafb9e55214818913a6f1b37a96d99 P 0d6b329f31654a509020ee35cac7b2b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:02:06.326323 10087 ts_tablet_manager.cc:1918] T c7dafb9e55214818913a6f1b37a96d99 P a28aec76a3204f7daf88d19d7781d18f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:02:06.326488 10087 log.cc:1196] T c7dafb9e55214818913a6f1b37a96d99 P a28aec76a3204f7daf88d19d7781d18f: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-1/wal/wals/c7dafb9e55214818913a6f1b37a96d99
I20260925 05:02:06.328292 10088 ts_tablet_manager.cc:1905] T c7dafb9e55214818913a6f1b37a96d99 P 0d6b329f31654a509020ee35cac7b2b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:02:06.329083 10087 ts_tablet_manager.cc:1939] T c7dafb9e55214818913a6f1b37a96d99 P a28aec76a3204f7daf88d19d7781d18f: Deleting consensus metadata
Sep 25 05:02:06 dist-test-slave-5n86 krb5kdc[9545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.1@KRBTEST.COM
I20260925 05:02:06.331017 9575 catalog_manager.cc:4841] TS a28aec76a3204f7daf88d19d7781d18f (127.30.247.2:44075): tablet c7dafb9e55214818913a6f1b37a96d99 (table test-table [id=165c52b7fc2d46fd9ce2dabd6697290c]) successfully deleted
I20260925 05:02:06.336107 10088 ts_tablet_manager.cc:1918] T c7dafb9e55214818913a6f1b37a96d99 P 0d6b329f31654a509020ee35cac7b2b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:02:06.336280 10088 log.cc:1196] T c7dafb9e55214818913a6f1b37a96d99 P 0d6b329f31654a509020ee35cac7b2b9: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312430481417-31708-0/minicluster-data/ts-0/wal/wals/c7dafb9e55214818913a6f1b37a96d99
I20260925 05:02:06.336936 10088 ts_tablet_manager.cc:1939] T c7dafb9e55214818913a6f1b37a96d99 P 0d6b329f31654a509020ee35cac7b2b9: Deleting consensus metadata
I20260925 05:02:06.338253 10062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:06.301234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32838 (local address 127.30.247.1:36717)
0925 05:02:06.301395 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:06.301403 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:06.311026 (+ 9623us) server_negotiation.cc:407] Connection header received
0925 05:02:06.311425 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:06.311430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:06.311516 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:06.311589 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:06.313337 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.314514 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.324589 (+ 10075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.325329 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.325429 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:06.330838 (+ 5409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:02:06.330851 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:02:06.330866 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:02:06.330905 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:02:06.334821 (+ 3916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:06.335609 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:06.335618 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:06.335622 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:06.335797 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:06.336264 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:06.336275 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:06.336279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:06.336772 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:02:06.337092 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:06.337443 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 05:02:06.337671 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 05:02:06.338802 9576 catalog_manager.cc:4841] TS 0d6b329f31654a509020ee35cac7b2b9 (127.30.247.1:36717): tablet c7dafb9e55214818913a6f1b37a96d99 (table test-table [id=165c52b7fc2d46fd9ce2dabd6697290c]) successfully deleted
I20260925 05:02:06.338899 9774 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:02:06.459777 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 9651
I20260925 05:02:06.477283 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 9785
Sep 25 05:02:06 dist-test-slave-5n86 krb5kdc[9545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.2@KRBTEST.COM
I20260925 05:02:06.494079 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:06.478262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60646 (local address 127.30.247.2:44075)
0925 05:02:06.478467 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:06.478474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:06.478496 (+ 22us) server_negotiation.cc:407] Connection header received
0925 05:02:06.478561 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:06.478566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:06.478647 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:06.478768 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:06.479879 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.481035 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.482470 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.483094 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.483218 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:06.487451 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:02:06.487468 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:02:06.487489 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:02:06.487537 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:02:06.491285 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:06.491927 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:06.491939 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:06.491942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:06.492085 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:06.492446 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:06.492451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:06.492455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:06.492959 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:02:06.493241 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:06.493494 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 05:02:06.493699 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 05:02:06.494768 9908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:02:06.583801 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 9785
I20260925 05:02:06.600667 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 9918
Sep 25 05:02:06 dist-test-slave-5n86 krb5kdc[9545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.3@KRBTEST.COM
I20260925 05:02:06.617588 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:06.601664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41612 (local address 127.30.247.3:46605)
0925 05:02:06.601884 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:06.601891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:06.601911 (+ 20us) server_negotiation.cc:407] Connection header received
0925 05:02:06.601978 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:06.601984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:06.602085 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:06.602208 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:06.603498 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.604753 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.606011 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:06.606674 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:06.606821 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:06.611346 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:02:06.611357 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:02:06.611371 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:02:06.611409 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:02:06.614771 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:06.615560 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:06.615566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:06.615568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:06.615716 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:06.616102 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:06.616105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:06.616107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:06.616439 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:02:06.616725 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:06.617059 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 05:02:06.617210 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 05:02:06.618247 10041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:02:06.735391 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 9918
I20260925 05:02:06.754410 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 9558
I20260925 05:02:06.755549 9620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:02:06.861320 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 9558
2026-09-25T05:02:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4154 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:02:06 dist-test-slave-5n86 krb5kdc[10096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:02:06 dist-test-slave-5n86 krb5kdc[10096](info): set up 2 sockets
Sep 25 05:02:06 dist-test-slave-5n86 krb5kdc[10096](info): commencing operation
krb5kdc: starting...
W20260925 05:02:07.941728 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:02:08 dist-test-slave-5n86 krb5kdc[10096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:02:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:02:08Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.30.247.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.30.247.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.30.247.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/rangeradmin_127.30.247.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.30.247.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/rangeradmin_127.30.247.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.30.247.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.30.247.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.30.247.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/rangerlookup_127.30.247.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.30.247.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/rangerlookup_127.30.247.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.247.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.30.247.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.247.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/HTTP_127.30.247.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.247.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/HTTP_127.30.247.20@KRBTEST.COM.keytab.
I20260925 05:02:08.236904 31708 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task43xS0j/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/postgres -l logfile start
2026-09-25 05:02:10.806 UTC [10126] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 05:02:10.806 UTC [10126] LOG: listening on IPv4 address "127.30.247.20", port 58211
2026-09-25 05:02:10.809 UTC [10126] LOG: listening on Unix socket "/tmp/.s.PGSQL.58211"
2026-09-25 05:02:10.830 UTC [10129] LOG: database system was shut down at 2026-09-25 05:02:09 UTC
2026-09-25 05:02:10.834 UTC [10126] LOG: database system is ready to accept connections
I20260925 05:02:11.809398 31708 mini_postgres.cc:96] Postgres bound to 58211
2026-09-25 05:02:11.826 UTC [10137] FATAL: database "slave" does not exist
127.30.247.20:58211 - accepting connections
I20260925 05:02:11.826756 31708 mini_ranger.cc:161] Starting Ranger...
I20260925 05:02:11.850965 31708 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 05:02:12.346755 31708 mini_ranger.cc:87] Created ranger Postgres database
I20260925 05:02:12.346868 31708 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ranger-admin
2026-09-25 05:02:12,627 [I] DB FLAVOR :POSTGRES
2026-09-25 05:02:12,627 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 05:02:12,627 [I] Checking connection..
2026-09-25 05:02:12,628 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 05:02:12,883 [I] Checking connection passed.
2026-09-25 05:02:12,883 [I] --------- Verifying version history table ---------
2026-09-25 05:02:12,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 05:02:13,119 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 05:02:13,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 05:02:13,369 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 05:02:13,369 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 05:02:13,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 05:02:13.608 UTC [10201] WARNING: there is no transaction in progress
2026-09-25 05:02:13,614 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 05:02:13,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 05:02:13,855 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 05:02:13,855 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 05:02:13,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 05:02:14,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 05:02:14,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5n86.c.gcp-upstream.internal','N') ;"
2026-09-25 05:02:14,541 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 05:02:14,784 [I] Table x_portal_user does not exist in database ranger
2026-09-25 05:02:14,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 05:02:15,036 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 05:02:15,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 05:02:15,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 05:02:15,516 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 05:02:15,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 05:02:16.469 UTC [10309] WARNING: there is no transaction in progress
2026-09-25 05:02:16.485 UTC [10309] WARNING: there is no transaction in progress
2026-09-25 05:02:16.501 UTC [10309] WARNING: there is no transaction in progress
2026-09-25 05:02:16.517 UTC [10309] WARNING: there is no transaction in progress
2026-09-25 05:02:16.533 UTC [10309] WARNING: there is no transaction in progress
2026-09-25 05:02:16.550 UTC [10309] WARNING: there is no transaction in progress
2026-09-25 05:02:16.606 UTC [10309] WARNING: there is no transaction in progress
2026-09-25 05:02:16.613 UTC [10309] WARNING: there is no transaction in progress
2026-09-25 05:02:16.628 UTC [10309] WARNING: there is no transaction in progress
2026-09-25 05:02:16.638 UTC [10309] WARNING: there is no transaction in progress
2026-09-25 05:02:16.648 UTC [10309] WARNING: there is no transaction in progress
2026-09-25 05:02:16.661 UTC [10309] WARNING: there is no transaction in progress
2026-09-25 05:02:16.670 UTC [10309] WARNING: there is no transaction in progress
2026-09-25 05:02:16.676 UTC [10309] WARNING: there is no transaction in progress
2026-09-25 05:02:16.683 UTC [10309] WARNING: there is no transaction in progress
2026-09-25 05:02:16.690 UTC [10309] WARNING: there is no transaction in progress
2026-09-25 05:02:17,332 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 05:02:17,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 05:02:17,561 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 05:02:17,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 05:02:17,799 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 05:02:17,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 05:02:18,051 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 05:02:18,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 05:02:18,277 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 05:02:18,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5n86.c.gcp-upstream.internal';"
2026-09-25 05:02:18,725 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 05:02:18,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 05:02:18,978 [I] DB_PATCHES have already been applied
I20260925 05:02:18.984440 31708 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ranger-admin:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task43xS0j/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 05:02:18.984521 31708 mini_ranger.cc:193] Using host: 127.30.247.20
I20260925 05:02:18.998502 31708 mini_ranger.cc:227] Ranger admin URL: http://127.30.247.20:47869
Sep 25, 2026 5:02:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 5:02:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 5:02:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 5:02:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 5:02:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 5:02:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 5:02:19 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 05:02:20.124357 31708 mini_ranger.cc:160] Time spent starting Ranger: real 8.298s user 0.000s sys 0.033s
Sep 25, 2026 5:02:39 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 5:02:39 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 5:02:39 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 5:02:39 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 5:02:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 5:02:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 5:02:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 5:02:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 5:02:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 5:02:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 5:02:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 5:02:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 5:02:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 5:02:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 5:02:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 5:02:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 5:02:39 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 5:02:40 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 05:02:40.869994 31708 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.30.247.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.30.247.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.30.247.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/rangerkms_127.30.247.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.30.247.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/rangerkms_127.30.247.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.247.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.30.247.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.247.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/HTTP_127.30.247.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.247.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/HTTP_127.30.247.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 05:02:41 dist-test-slave-5n86 krb5kdc[10096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312561, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 05:02:41.030184 31708 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 05:02:41.047623 31708 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 05:02:41.964463 31708 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 05:02:41.964677 31708 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ranger-kms
I20260925 05:02:41.964819 31708 mini_ranger_kms.cc:227] Using postgres at 127.30.247.20:58211
2026-09-25 05:02:42,396 [I] DB FLAVOR :POSTGRES
2026-09-25 05:02:42,396 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 05:02:42,396 [I] Checking connection
2026-09-25 05:02:42,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:58211/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 05:02:42,633 [I] connection success
2026-09-25 05:02:42,633 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 05:02:42,633 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 05:02:42,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:58211/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 05:02:42,853 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 05:02:42,853 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 05:02:42,853 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 05:02:42,853 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:58211/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 05:02:43,107 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 05:02:43.568418 31708 mini_ranger_kms.cc:310] Created kms service
I20260925 05:02:43.751852 31708 mini_ranger_kms.cc:326] Created kudu user
I20260925 05:02:43.794859 31708 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 05:02:43 dist-test-slave-5n86 krb5kdc[10096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312561, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.30.247.20@KRBTEST.COM
I20260925 05:02:44.188647 31708 mini_ranger_kms.cc:384] Added ranger policy
I20260925 05:02:44.188978 31708 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ranger-kms:/tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 05:02:44.189021 31708 mini_ranger_kms.cc:240] Using host: 127.30.247.20
I20260925 05:02:44.209789 31708 mini_ranger_kms.cc:276] Ranger KMS PID: 10489
I20260925 05:02:44.209966 31708 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.30.247.20:48381
Sep 25, 2026 5:02:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 5:02:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 5:02:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 5:02:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 5:02:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 5:02:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 05:02:46.466711 31708 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.437s user 0.121s sys 0.321s
I20260925 05:02:46.466961 31708 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 05:02:46.467185 31708 mini_ranger_kms.cc:401] 127.30.247.20:48381/kms/v1/keys
Sep 25 05:02:46 dist-test-slave-5n86 krb5kdc[10096](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790312566, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.247.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 05:02:46 dist-test-slave-5n86 krb5kdc[10096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312566, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.247.20@KRBTEST.COM for HTTP/127.30.247.20@KRBTEST.COM
Sep 25 05:02:47 dist-test-slave-5n86 krb5kdc[10096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312566, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.247.20@KRBTEST.COM for HTTP/127.30.247.20@KRBTEST.COM
Sep 25 05:02:49 dist-test-slave-5n86 krb5kdc[10096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:02:49.460047 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:41507
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:41507
--ranger_config_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.247.20:48381/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:02:49.781376 10540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:02:49.781701 10540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:02:49.781852 10540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:02:49.788691 10540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:02:49.788811 10540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:02:49.788903 10540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:02:49.789001 10540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:02:49.789088 10540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:02:49.798543 10540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/master-0/wal
--ranger_kms_url=127.30.247.20:48381/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:41507
--ranger_config_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:41507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:02:49.799708 10540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:02:49.800973 10540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:02:49.813935 10545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:02:49.814049 10548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:02:49.814072 10546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:02:49.815114 10540 server_base.cc:1034] running on GCE node
I20260925 05:02:49.815889 10540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:02:49.817301 10540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:02:49.818558 10540 hybrid_clock.cc:648] HybridClock initialized: now 1790312569818516 us; error 35 us; skew 500 ppm
I20260925 05:02:49.818933 10540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:02:49 dist-test-slave-5n86 krb5kdc[10096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312569, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:02:49.825314 10540 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:02:49.826959 10540 webserver.cc:460] Webserver started at http://127.30.247.62:33795/ using document root <none> and password file <none>
I20260925 05:02:49.827713 10540 fs_manager.cc:359] Metadata directory not provided
I20260925 05:02:49.827834 10540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:02:49.828132 10540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:02:49 dist-test-slave-5n86 krb5kdc[10096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312569, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for HTTP/127.30.247.20@KRBTEST.COM
I20260925 05:02:49.883818 10540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "7e34710f690142e0b1320218be91924a"
format_stamp: "Formatted at 2026-09-25 05:02:49 on dist-test-slave-5n86"
server_key: "83621e52f216ca1adaa22dcce2fa2908"
server_key_iv: "8a7029986326c5030598e1742e11a92b"
server_key_version: "kuduclusterkey@0"
I20260925 05:02:49.884732 10540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "7e34710f690142e0b1320218be91924a"
format_stamp: "Formatted at 2026-09-25 05:02:49 on dist-test-slave-5n86"
server_key: "83621e52f216ca1adaa22dcce2fa2908"
server_key_iv: "8a7029986326c5030598e1742e11a92b"
server_key_version: "kuduclusterkey@0"
I20260925 05:02:49.890182 10540 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 05:02:49.905426 10555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:49.906874 10540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:02:49.907039 10540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "7e34710f690142e0b1320218be91924a"
format_stamp: "Formatted at 2026-09-25 05:02:49 on dist-test-slave-5n86"
server_key: "83621e52f216ca1adaa22dcce2fa2908"
server_key_iv: "8a7029986326c5030598e1742e11a92b"
server_key_version: "kuduclusterkey@0"
I20260925 05:02:49.907238 10540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:02:49.922760 10540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:02:49.926533 10540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:02:49.926836 10540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:02:49.946045 10540 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:41507
I20260925 05:02:49.946064 10606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:41507 every 8 connection(s)
I20260925 05:02:49.947566 10540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:02:49.951242 10607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:02:49.954605 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 10540
I20260925 05:02:49.954807 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:02:49.955569 31708 external_mini_cluster.cc:1428] Setting key a9483478d83ce030f08807e6c8d00322
I20260925 05:02:49.970295 10607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e34710f690142e0b1320218be91924a: Bootstrap starting.
I20260925 05:02:49.975340 10607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e34710f690142e0b1320218be91924a: Neither blocks nor log segments found. Creating new log.
I20260925 05:02:49.977365 10607 log.cc:824] T 00000000000000000000000000000000 P 7e34710f690142e0b1320218be91924a: Log is configured to *not* fsync() on all Append() calls
I20260925 05:02:49.980926 10607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e34710f690142e0b1320218be91924a: No bootstrap required, opened a new log
Sep 25 05:02:49 dist-test-slave-5n86 krb5kdc[10096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:02:49.991518 10607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e34710f690142e0b1320218be91924a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e34710f690142e0b1320218be91924a" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 41507 } }
I20260925 05:02:49.991966 10607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e34710f690142e0b1320218be91924a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:02:49.992075 10607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e34710f690142e0b1320218be91924a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e34710f690142e0b1320218be91924a, State: Initialized, Role: FOLLOWER
I20260925 05:02:49.992820 10607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e34710f690142e0b1320218be91924a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e34710f690142e0b1320218be91924a" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 41507 } }
I20260925 05:02:49.993096 10607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e34710f690142e0b1320218be91924a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:02:49.993234 10607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e34710f690142e0b1320218be91924a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:02:49.993409 10607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e34710f690142e0b1320218be91924a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:02:49.996133 10610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:49.957632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45862 (local address 127.30.247.62:41507)
0925 05:02:49.976304 (+ 18672us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:49.976339 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:49.976992 (+ 653us) server_negotiation.cc:407] Connection header received
0925 05:02:49.979329 (+ 2337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:49.979351 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:49.979903 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:49.980270 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:49.981617 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:49.982876 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:49.984100 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:49.984841 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:49.985027 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:49.988941 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:02:49.988957 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:02:49.988981 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:02:49.989037 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:02:49.992225 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:49.992917 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:49.992925 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:49.992933 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:49.993115 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:49.993568 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:49.993573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:49.993577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:49.994071 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:02:49.994367 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:49.994839 (+ 472us) server_negotiation.cc:299] Negotiation successful
0925 05:02:49.995214 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18350,"thread_start_us":18119,"threads_started":1}
I20260925 05:02:49.996749 10607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e34710f690142e0b1320218be91924a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e34710f690142e0b1320218be91924a" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 41507 } }
I20260925 05:02:49.997308 10607 leader_election.cc:302] T 00000000000000000000000000000000 P 7e34710f690142e0b1320218be91924a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e34710f690142e0b1320218be91924a; no voters:
I20260925 05:02:49.998019 10607 leader_election.cc:288] T 00000000000000000000000000000000 P 7e34710f690142e0b1320218be91924a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:02:49.998111 10612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e34710f690142e0b1320218be91924a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:02:50.000859 10612 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e34710f690142e0b1320218be91924a [term 1 LEADER]: Becoming Leader. State: Replica: 7e34710f690142e0b1320218be91924a, State: Running, Role: LEADER
I20260925 05:02:50.001441 10612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e34710f690142e0b1320218be91924a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e34710f690142e0b1320218be91924a" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 41507 } }
I20260925 05:02:50.008025 10614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e34710f690142e0b1320218be91924a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e34710f690142e0b1320218be91924a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e34710f690142e0b1320218be91924a" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 41507 } } }
I20260925 05:02:50.008431 10614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e34710f690142e0b1320218be91924a [sys.catalog]: This master's current role is: LEADER
I20260925 05:02:50.011082 10613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e34710f690142e0b1320218be91924a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e34710f690142e0b1320218be91924a. Latest consensus state: current_term: 1 leader_uuid: "7e34710f690142e0b1320218be91924a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e34710f690142e0b1320218be91924a" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 41507 } } }
I20260925 05:02:50.011729 10613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e34710f690142e0b1320218be91924a [sys.catalog]: This master's current role is: LEADER
I20260925 05:02:50.014817 10607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e34710f690142e0b1320218be91924a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:02:50.016021 10616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:02:50.021821 10616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:02:50.038779 10616 catalog_manager.cc:1347] Generated new cluster ID: bcaf9ceb61674a27af190b23a58fd8fb
I20260925 05:02:50.039049 10616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:02:50.068142 10616 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:02:50.070606 10616 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:02:50.080879 10616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e34710f690142e0b1320218be91924a: Generated new TSK 0
I20260925 05:02:50.082135 10616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:02:50.089273 10607 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
WARNING: no policy specified for kudu/127.30.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
I20260925 05:02:50.934185 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.247.20:48381/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:0
--local_ip_for_outbound_sockets=127.30.247.1
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:41507
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:02:51 dist-test-slave-5n86 krb5kdc[10096](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790312571, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 05:02:51.376237 10653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:02:51.376739 10653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:02:51.376909 10653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:02:51.386296 10653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:02:51.386399 10653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:02:51.386588 10653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:02:51.396314 10653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.30.247.20:48381/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:41507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:02:51.397650 10653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:02:51.398967 10653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:02:51.412868 10664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:02:51.413345 10663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:02:51.414816 10653 server_base.cc:1034] running on GCE node
W20260925 05:02:51.415563 10666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:02:51.416330 10653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:02:51.417388 10653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:02:51.418735 10653 hybrid_clock.cc:648] HybridClock initialized: now 1790312571418667 us; error 52 us; skew 500 ppm
I20260925 05:02:51.419183 10653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:02:51 dist-test-slave-5n86 krb5kdc[10096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312571, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:02:51.422645 10653 init.cc:377] Logged in from keytab as kudu/127.30.247.1@KRBTEST.COM (short username kudu)
I20260925 05:02:51.434624 10653 webserver.cc:460] Webserver started at http://127.30.247.1:35191/ using document root <none> and password file <none>
I20260925 05:02:51.435357 10653 fs_manager.cc:359] Metadata directory not provided
I20260925 05:02:51.435503 10653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:02:51.435822 10653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:02:51 dist-test-slave-5n86 krb5kdc[10096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312571, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for HTTP/127.30.247.20@KRBTEST.COM
I20260925 05:02:51.459098 10653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-0/data/instance:
uuid: "f9a0ef2e4f8d474fa81d6d8ed5869a23"
format_stamp: "Formatted at 2026-09-25 05:02:51 on dist-test-slave-5n86"
server_key: "1b0690c38e104d4b4d7715fb23d31ca1"
server_key_iv: "d46b112fe391707db176fa93a3d5c79e"
server_key_version: "kuduclusterkey@0"
I20260925 05:02:51.460122 10653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance:
uuid: "f9a0ef2e4f8d474fa81d6d8ed5869a23"
format_stamp: "Formatted at 2026-09-25 05:02:51 on dist-test-slave-5n86"
server_key: "1b0690c38e104d4b4d7715fb23d31ca1"
server_key_iv: "d46b112fe391707db176fa93a3d5c79e"
server_key_version: "kuduclusterkey@0"
I20260925 05:02:51.478504 10653 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.020s sys 0.000s
I20260925 05:02:51.510212 10673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:51.516973 10653 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.003s sys 0.005s
I20260925 05:02:51.517184 10653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "f9a0ef2e4f8d474fa81d6d8ed5869a23"
format_stamp: "Formatted at 2026-09-25 05:02:51 on dist-test-slave-5n86"
server_key: "1b0690c38e104d4b4d7715fb23d31ca1"
server_key_iv: "d46b112fe391707db176fa93a3d5c79e"
server_key_version: "kuduclusterkey@0"
I20260925 05:02:51.517473 10653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:02:51.555294 10653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:02:51.560248 10653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:02:51.560633 10653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:02:51.561753 10653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:02:51.563530 10653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:02:51.563637 10653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:51.563766 10653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:02:51.563833 10653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:51.629601 10653 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:43427
I20260925 05:02:51.629613 10785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:43427 every 8 connection(s)
Sep 25 05:02:51 dist-test-slave-5n86 krb5kdc[10096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312571, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for HTTP/127.30.247.20@KRBTEST.COM
I20260925 05:02:51.631357 10653 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:02:51.634076 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 10653
I20260925 05:02:51.634272 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance
I20260925 05:02:51.635007 31708 external_mini_cluster.cc:1428] Setting key 312cbae9a43a6761675d3fd109f9368b
Sep 25 05:02:51 dist-test-slave-5n86 krb5kdc[10096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312571, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.247.2@KRBTEST.COM; defaulting to no policy
I20260925 05:02:51.689970 10789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:51.637768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:41507 (local address 127.30.247.1:37589)
0925 05:02:51.645258 (+ 7490us) negotiation.cc:107] Waiting for socket to connect
0925 05:02:51.645321 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 05:02:51.648138 (+ 2817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:02:51.659723 (+ 11585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:02:51.659741 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:02:51.660561 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:02:51.661889 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:51.661924 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:51.663592 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:51.663601 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:51.665667 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:51.665683 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:51.666948 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:51.666956 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:51.667253 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:51.668228 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:02:51.668260 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:02:51.671104 (+ 2844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:02:51.684807 (+ 13703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:02:51.684818 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:02:51.684840 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:02:51.685484 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:02:51.686253 (+ 769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:02:51.686258 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:02:51.686261 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:02:51.686555 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:02:51.687215 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:02:51.687224 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:02:51.687671 (+ 447us) client_negotiation.cc:769] Sending connection context
0925 05:02:51.687975 (+ 304us) client_negotiation.cc:240] Negotiation successful
0925 05:02:51.688525 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7085,"thread_start_us":9226,"threads_started":1}
I20260925 05:02:51.692519 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:51.637565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:37589 (local address 127.30.247.62:41507)
0925 05:02:51.658923 (+ 21358us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:51.658933 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:51.658966 (+ 33us) server_negotiation.cc:407] Connection header received
0925 05:02:51.659059 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:51.659067 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:51.659200 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:51.659367 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:51.662159 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:51.663399 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:51.665882 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:51.666500 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:51.666630 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:51.671369 (+ 4739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:02:51.671378 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:02:51.671386 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:02:51.671441 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:02:51.684583 (+ 13142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:51.685949 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:51.685955 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:51.685958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:51.686112 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:51.686716 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:51.686721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:51.686725 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:51.687061 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:02:51.691804 (+ 4743us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:51.691932 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 05:02:51.692157 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21119,"thread_start_us":21965,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.2@KRBTEST.COM" created.
I20260925 05:02:51.694392 10786 heartbeater.cc:347] Connected to a master server at 127.30.247.62:41507
I20260925 05:02:51.694726 10786 heartbeater.cc:464] Registering TS with master...
I20260925 05:02:51.695550 10786 heartbeater.cc:511] Master 127.30.247.62:41507 requested a full tablet report, sending...
I20260925 05:02:51.698119 10572 ts_manager.cc:194] Registered new tserver with Master: f9a0ef2e4f8d474fa81d6d8ed5869a23 (127.30.247.1:43427)
I20260925 05:02:51.700215 10572 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.1@KRBTEST.COM'} at 127.30.247.1:37589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
I20260925 05:02:51.728456 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.247.20:48381/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:0
--local_ip_for_outbound_sockets=127.30.247.2
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:41507
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:02:51 dist-test-slave-5n86 krb5kdc[10096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312571, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for HTTP/127.30.247.20@KRBTEST.COM
W20260925 05:02:52.167543 10792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:02:52.168048 10792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:02:52.168254 10792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:02:52.179749 10792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:02:52.179903 10792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:02:52.180089 10792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:02:52.191418 10792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.30.247.20:48381/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:41507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:02:52.193457 10792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:02:52.195271 10792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:02:52.213735 10800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:02:52.213464 10799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:02:52.214696 10792 server_base.cc:1034] running on GCE node
W20260925 05:02:52.219340 10802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:02:52.220136 10792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:02:52.228394 10792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:02:52.229658 10792 hybrid_clock.cc:648] HybridClock initialized: now 1790312572229550 us; error 87 us; skew 500 ppm
I20260925 05:02:52.230067 10792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:02:52 dist-test-slave-5n86 krb5kdc[10096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312572, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:02:52.234295 10792 init.cc:377] Logged in from keytab as kudu/127.30.247.2@KRBTEST.COM (short username kudu)
I20260925 05:02:52.243768 10792 webserver.cc:460] Webserver started at http://127.30.247.2:45599/ using document root <none> and password file <none>
I20260925 05:02:52.244656 10792 fs_manager.cc:359] Metadata directory not provided
I20260925 05:02:52.244786 10792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:02:52.245131 10792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:02:52 dist-test-slave-5n86 krb5kdc[10096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312572, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for HTTP/127.30.247.20@KRBTEST.COM
I20260925 05:02:52.259861 10792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-1/data/instance:
uuid: "3aaeb7323d1548299d7e4031bc1be3b4"
format_stamp: "Formatted at 2026-09-25 05:02:52 on dist-test-slave-5n86"
server_key: "bbc0ca75b35fd004d184aa609cfd4cc3"
server_key_iv: "346f9bec8b0db6d9d4c7b1f2e7bf0457"
server_key_version: "kuduclusterkey@0"
I20260925 05:02:52.260730 10792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance:
uuid: "3aaeb7323d1548299d7e4031bc1be3b4"
format_stamp: "Formatted at 2026-09-25 05:02:52 on dist-test-slave-5n86"
server_key: "bbc0ca75b35fd004d184aa609cfd4cc3"
server_key_iv: "346f9bec8b0db6d9d4c7b1f2e7bf0457"
server_key_version: "kuduclusterkey@0"
I20260925 05:02:52.265797 10792 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 05:02:52.279727 10816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:52.281213 10792 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:02:52.281380 10792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "3aaeb7323d1548299d7e4031bc1be3b4"
format_stamp: "Formatted at 2026-09-25 05:02:52 on dist-test-slave-5n86"
server_key: "bbc0ca75b35fd004d184aa609cfd4cc3"
server_key_iv: "346f9bec8b0db6d9d4c7b1f2e7bf0457"
server_key_version: "kuduclusterkey@0"
I20260925 05:02:52.281584 10792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:02:52.295910 10792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:02:52.299542 10792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:02:52.299830 10792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:02:52.300798 10792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:02:52.302191 10792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:02:52.302287 10792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:52.302425 10792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:02:52.302495 10792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:52.339805 10792 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:36313
I20260925 05:02:52.339823 10928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:36313 every 8 connection(s)
I20260925 05:02:52.341172 10792 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:02:52.350260 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 10792
I20260925 05:02:52.350431 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance
I20260925 05:02:52.350965 31708 external_mini_cluster.cc:1428] Setting key 91eae05f9975fa2efbae804ab6d766e9
Sep 25 05:02:52 dist-test-slave-5n86 krb5kdc[10096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312572, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:02:52.365716 10931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:52.343946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:36267 (local address 127.30.247.62:41507)
0925 05:02:52.344441 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:52.344445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:52.345486 (+ 1041us) server_negotiation.cc:407] Connection header received
0925 05:02:52.347219 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:52.347225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:52.347326 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:52.347443 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:52.348864 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:52.349589 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:52.351324 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:52.351859 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:52.351988 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:52.359903 (+ 7915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:02:52.359908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:02:52.359911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:02:52.359941 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:02:52.362576 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:52.363382 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:52.363388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:52.363393 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:52.363510 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:52.363918 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:52.363923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:52.363926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:52.364211 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:02:52.364377 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:52.365118 (+ 741us) server_negotiation.cc:299] Negotiation successful
0925 05:02:52.365335 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":276,"threads_started":1}
I20260925 05:02:52.366742 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:52.344574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:41507 (local address 127.30.247.2:36267)
0925 05:02:52.345332 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0925 05:02:52.345381 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 05:02:52.346955 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:02:52.347671 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:02:52.347678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:02:52.348074 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:02:52.348708 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:52.348726 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:52.349771 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:52.349776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:52.351178 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:52.351190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:52.352084 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:52.352094 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:52.352334 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:52.356742 (+ 4408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:02:52.356776 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:02:52.359691 (+ 2915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:02:52.362760 (+ 3069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:02:52.362769 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:02:52.362787 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:02:52.363257 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:02:52.363619 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:02:52.363624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:02:52.363628 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:02:52.363801 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:02:52.364346 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:02:52.364354 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:02:52.364760 (+ 406us) client_negotiation.cc:769] Sending connection context
0925 05:02:52.365016 (+ 256us) client_negotiation.cc:240] Negotiation successful
0925 05:02:52.365530 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":305,"threads_started":1}
I20260925 05:02:52.368168 10929 heartbeater.cc:347] Connected to a master server at 127.30.247.62:41507
I20260925 05:02:52.368510 10929 heartbeater.cc:464] Registering TS with master...
I20260925 05:02:52.369249 10929 heartbeater.cc:511] Master 127.30.247.62:41507 requested a full tablet report, sending...
I20260925 05:02:52.370644 10572 ts_manager.cc:194] Registered new tserver with Master: 3aaeb7323d1548299d7e4031bc1be3b4 (127.30.247.2:36313)
I20260925 05:02:52.371519 10572 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.2@KRBTEST.COM'} at 127.30.247.2:36267
WARNING: no policy specified for kudu/127.30.247.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
I20260925 05:02:52.408826 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.247.20:48381/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:0
--local_ip_for_outbound_sockets=127.30.247.3
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:41507
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:02:52.703980 10786 heartbeater.cc:503] Master 127.30.247.62:41507 was elected leader, sending a full tablet report...
W20260925 05:02:52.728098 10935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:02:52.728384 10935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:02:52.728561 10935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:02:52.735471 10935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:02:52.735566 10935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:02:52.735672 10935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:02:52.745002 10935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.30.247.20:48381/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:41507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:02:52.746181 10935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:02:52.747391 10935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:02:52.759493 10940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:02:52.759516 10941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:02:52.759807 10943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:02:52.760703 10935 server_base.cc:1034] running on GCE node
I20260925 05:02:52.761248 10935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:02:52.762152 10935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:02:52.763401 10935 hybrid_clock.cc:648] HybridClock initialized: now 1790312572763361 us; error 28 us; skew 500 ppm
I20260925 05:02:52.763748 10935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:02:52 dist-test-slave-5n86 krb5kdc[10096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312572, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:02:52.766784 10935 init.cc:377] Logged in from keytab as kudu/127.30.247.3@KRBTEST.COM (short username kudu)
I20260925 05:02:52.768149 10935 webserver.cc:460] Webserver started at http://127.30.247.3:33765/ using document root <none> and password file <none>
I20260925 05:02:52.768858 10935 fs_manager.cc:359] Metadata directory not provided
I20260925 05:02:52.769003 10935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:02:52.769296 10935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:02:52 dist-test-slave-5n86 krb5kdc[10096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312572, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for HTTP/127.30.247.20@KRBTEST.COM
I20260925 05:02:52.782738 10935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-2/data/instance:
uuid: "dfa792f2c0a64e309c30a8cca705f55f"
format_stamp: "Formatted at 2026-09-25 05:02:52 on dist-test-slave-5n86"
server_key: "a490862af4bd03dbcbc510fea4bfa44a"
server_key_iv: "0d97d0080e3355009fc5d468b0a7f227"
server_key_version: "kuduclusterkey@0"
I20260925 05:02:52.783500 10935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance:
uuid: "dfa792f2c0a64e309c30a8cca705f55f"
format_stamp: "Formatted at 2026-09-25 05:02:52 on dist-test-slave-5n86"
server_key: "a490862af4bd03dbcbc510fea4bfa44a"
server_key_iv: "0d97d0080e3355009fc5d468b0a7f227"
server_key_version: "kuduclusterkey@0"
I20260925 05:02:52.788707 10935 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260925 05:02:52.803570 10950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:52.805007 10935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:02:52.805169 10935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "dfa792f2c0a64e309c30a8cca705f55f"
format_stamp: "Formatted at 2026-09-25 05:02:52 on dist-test-slave-5n86"
server_key: "a490862af4bd03dbcbc510fea4bfa44a"
server_key_iv: "0d97d0080e3355009fc5d468b0a7f227"
server_key_version: "kuduclusterkey@0"
I20260925 05:02:52.805377 10935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:02:52.822903 10935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:02:52.826747 10935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:02:52.827035 10935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:02:52.827878 10935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:02:52.829236 10935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:02:52.829308 10935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:52.829465 10935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:02:52.829514 10935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:02:52.859629 10935 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:35371
I20260925 05:02:52.859668 11062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:35371 every 8 connection(s)
I20260925 05:02:52.860998 10935 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:02:52.869903 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 10935
I20260925 05:02:52.870082 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance
I20260925 05:02:52.870622 31708 external_mini_cluster.cc:1428] Setting key 8ebaac00de9729f1e1ef3ad48e958e60
Sep 25 05:02:52 dist-test-slave-5n86 krb5kdc[10096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312572, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:02:52.880856 10931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:52.863687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:50977 (local address 127.30.247.62:41507)
0925 05:02:52.863894 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:52.863900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:52.865241 (+ 1341us) server_negotiation.cc:407] Connection header received
0925 05:02:52.866878 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:52.866883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:52.866960 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:52.867047 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:52.868345 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:52.869246 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:52.871407 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:52.871889 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:52.872050 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:52.875565 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:02:52.875570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:02:52.875573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:02:52.875603 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:02:52.878196 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:52.878902 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:52.878906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:52.878908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:52.879035 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:52.879405 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:52.879408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:52.879410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:52.879661 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:02:52.879782 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:52.880384 (+ 602us) server_negotiation.cc:299] Negotiation successful
0925 05:02:52.880595 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:02:52.881898 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:52.864364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:41507 (local address 127.30.247.3:50977)
0925 05:02:52.865086 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0925 05:02:52.865123 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:02:52.866629 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:02:52.867238 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:02:52.867246 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:02:52.867653 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:02:52.868176 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:52.868192 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:52.869389 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:52.869392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:52.871246 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:52.871257 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:52.872138 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:52.872145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:52.872462 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:52.873204 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:02:52.873232 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:02:52.875373 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:02:52.878326 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:02:52.878332 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:02:52.878343 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:02:52.878768 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:02:52.879143 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:02:52.879146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:02:52.879149 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:02:52.879311 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:02:52.879781 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:02:52.879786 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:02:52.880046 (+ 260us) client_negotiation.cc:769] Sending connection context
0925 05:02:52.880280 (+ 234us) client_negotiation.cc:240] Negotiation successful
0925 05:02:52.880699 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":336,"threads_started":1}
I20260925 05:02:52.883249 11063 heartbeater.cc:347] Connected to a master server at 127.30.247.62:41507
I20260925 05:02:52.883533 11063 heartbeater.cc:464] Registering TS with master...
I20260925 05:02:52.884251 11063 heartbeater.cc:511] Master 127.30.247.62:41507 requested a full tablet report, sending...
I20260925 05:02:52.886036 10572 ts_manager.cc:194] Registered new tserver with Master: dfa792f2c0a64e309c30a8cca705f55f (127.30.247.3:35371)
I20260925 05:02:52.886646 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:02:52.887105 10572 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.3@KRBTEST.COM'} at 127.30.247.3:50977
I20260925 05:02:52.904003 10931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:52.891646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45876 (local address 127.30.247.62:41507)
0925 05:02:52.891891 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:52.891898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:52.892218 (+ 320us) server_negotiation.cc:407] Connection header received
0925 05:02:52.892407 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:52.892417 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:52.892481 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:52.892608 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:52.893980 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:52.894689 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:52.895967 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:52.896385 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:52.896519 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:52.898821 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:02:52.898825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:02:52.898828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:02:52.898854 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:02:52.901484 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:52.902117 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:52.902124 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:52.902127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:52.902254 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:52.902613 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:52.902619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:52.902622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:52.902884 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:02:52.903154 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:52.903458 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 05:02:52.903654 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260925 05:02:52.907951 10572 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:02:52.915326 10572 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:02:52.953284 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:52.928054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:35371 (local address 127.0.0.1:43378)
0925 05:02:52.940395 (+ 12341us) negotiation.cc:107] Waiting for socket to connect
0925 05:02:52.940415 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:02:52.940518 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:02:52.942984 (+ 2466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:02:52.942992 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:02:52.943031 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:02:52.943527 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:52.943542 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:52.946077 (+ 2535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:52.946085 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:52.949309 (+ 3224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:52.949334 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:52.951924 (+ 2590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:52.951934 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:52.952203 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:52.952254 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 05:02:52.952332 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 05:02:52.952642 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12200,"mutex_wait_us":30,"thread_start_us":12061,"threads_started":1}
I20260925 05:02:52.953824 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:52.928118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32902 (local address 127.30.247.2:36313)
0925 05:02:52.940679 (+ 12561us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:52.940688 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:52.940727 (+ 39us) server_negotiation.cc:407] Connection header received
0925 05:02:52.940838 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:52.940849 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:52.941128 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:52.941333 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:02:52.942398 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:52.945534 (+ 3136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:52.949540 (+ 4006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:52.951078 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:52.951196 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:52.951394 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:52.953365 (+ 1971us) server_negotiation.cc:299] Negotiation successful
0925 05:02:52.953546 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12384,"thread_start_us":12437,"threads_started":1}
I20260925 05:02:52.956310 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:52.928616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43378 (local address 127.30.247.3:35371)
0925 05:02:52.942413 (+ 13797us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:52.942421 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:52.942452 (+ 31us) server_negotiation.cc:407] Connection header received
0925 05:02:52.942544 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:52.942551 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:52.942765 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:52.942951 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:02:52.943806 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:52.945876 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:52.949577 (+ 3701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:52.951607 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:52.955179 (+ 3572us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:52.955517 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:52.955602 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 05:02:52.955837 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13623,"thread_start_us":13444,"threads_started":1}
I20260925 05:02:52.956856 11076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:52.928024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:36313 (local address 127.0.0.1:32902)
0925 05:02:52.939707 (+ 11683us) negotiation.cc:107] Waiting for socket to connect
0925 05:02:52.939778 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 05:02:52.940025 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:02:52.941590 (+ 1565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:02:52.941598 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:02:52.941627 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:02:52.942202 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:52.942216 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:52.945687 (+ 3471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:52.945709 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:52.949371 (+ 3662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:52.949385 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:52.951525 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:52.951531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:52.951736 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:52.951783 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 05:02:52.952929 (+ 1146us) client_negotiation.cc:240] Negotiation successful
0925 05:02:52.953078 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11296,"mutex_wait_us":41,"thread_start_us":12804,"threads_started":1}
I20260925 05:02:52.961272 10864 tablet_service.cc:1467] Processing CreateTablet for tablet cb7772cae3df4a99b7ee7c9c7ea1afcc (DEFAULT_TABLE table=test-table [id=4c06e58583c242b9b0fbae927835a375]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:02:52.962726 10864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb7772cae3df4a99b7ee7c9c7ea1afcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:02:52.964323 10998 tablet_service.cc:1467] Processing CreateTablet for tablet cb7772cae3df4a99b7ee7c9c7ea1afcc (DEFAULT_TABLE table=test-table [id=4c06e58583c242b9b0fbae927835a375]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:02:52.966027 10998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb7772cae3df4a99b7ee7c9c7ea1afcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:02:52.974640 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:52.943307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:43427 (local address 127.0.0.1:52472)
0925 05:02:52.953574 (+ 10267us) negotiation.cc:107] Waiting for socket to connect
0925 05:02:52.953593 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:02:52.954930 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:02:52.967996 (+ 13066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:02:52.968003 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:02:52.968035 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:02:52.968219 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:52.968231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:52.970186 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:52.970192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:52.972028 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:52.972042 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:52.973871 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:52.973878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:52.974071 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:52.974106 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 05:02:52.974203 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 05:02:52.974344 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10151,"thread_start_us":13313,"threads_started":1}
I20260925 05:02:52.974764 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:52.941551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52472 (local address 127.30.247.1:43427)
0925 05:02:52.967388 (+ 25837us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:52.967396 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:52.967423 (+ 27us) server_negotiation.cc:407] Connection header received
0925 05:02:52.967516 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:52.967523 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:52.967749 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:52.967971 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:02:52.968390 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:52.970000 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:52.972189 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:52.973682 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:52.973814 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:52.974032 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:52.974271 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 05:02:52.974429 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25654,"thread_start_us":25509,"threads_started":1}
I20260925 05:02:52.978088 11082 tablet_bootstrap.cc:492] T cb7772cae3df4a99b7ee7c9c7ea1afcc P 3aaeb7323d1548299d7e4031bc1be3b4: Bootstrap starting.
I20260925 05:02:52.978430 10721 tablet_service.cc:1467] Processing CreateTablet for tablet cb7772cae3df4a99b7ee7c9c7ea1afcc (DEFAULT_TABLE table=test-table [id=4c06e58583c242b9b0fbae927835a375]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:02:52.979928 10721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb7772cae3df4a99b7ee7c9c7ea1afcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:02:52.980459 11081 tablet_bootstrap.cc:492] T cb7772cae3df4a99b7ee7c9c7ea1afcc P dfa792f2c0a64e309c30a8cca705f55f: Bootstrap starting.
I20260925 05:02:52.982012 11082 tablet_bootstrap.cc:654] T cb7772cae3df4a99b7ee7c9c7ea1afcc P 3aaeb7323d1548299d7e4031bc1be3b4: Neither blocks nor log segments found. Creating new log.
I20260925 05:02:52.982991 11082 log.cc:824] T cb7772cae3df4a99b7ee7c9c7ea1afcc P 3aaeb7323d1548299d7e4031bc1be3b4: Log is configured to *not* fsync() on all Append() calls
I20260925 05:02:52.983683 11081 tablet_bootstrap.cc:654] T cb7772cae3df4a99b7ee7c9c7ea1afcc P dfa792f2c0a64e309c30a8cca705f55f: Neither blocks nor log segments found. Creating new log.
I20260925 05:02:52.984902 11081 log.cc:824] T cb7772cae3df4a99b7ee7c9c7ea1afcc P dfa792f2c0a64e309c30a8cca705f55f: Log is configured to *not* fsync() on all Append() calls
I20260925 05:02:52.985931 11082 tablet_bootstrap.cc:492] T cb7772cae3df4a99b7ee7c9c7ea1afcc P 3aaeb7323d1548299d7e4031bc1be3b4: No bootstrap required, opened a new log
I20260925 05:02:52.986198 11082 ts_tablet_manager.cc:1397] T cb7772cae3df4a99b7ee7c9c7ea1afcc P 3aaeb7323d1548299d7e4031bc1be3b4: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 05:02:52.987723 11081 tablet_bootstrap.cc:492] T cb7772cae3df4a99b7ee7c9c7ea1afcc P dfa792f2c0a64e309c30a8cca705f55f: No bootstrap required, opened a new log
I20260925 05:02:52.987950 11081 ts_tablet_manager.cc:1397] T cb7772cae3df4a99b7ee7c9c7ea1afcc P dfa792f2c0a64e309c30a8cca705f55f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 05:02:52.993462 11082 raft_consensus.cc:348] T cb7772cae3df4a99b7ee7c9c7ea1afcc P 3aaeb7323d1548299d7e4031bc1be3b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aaeb7323d1548299d7e4031bc1be3b4" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36313 } } peers { permanent_uuid: "dfa792f2c0a64e309c30a8cca705f55f" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35371 } } peers { permanent_uuid: "f9a0ef2e4f8d474fa81d6d8ed5869a23" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 43427 } }
I20260925 05:02:52.993793 11082 raft_consensus.cc:374] T cb7772cae3df4a99b7ee7c9c7ea1afcc P 3aaeb7323d1548299d7e4031bc1be3b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:02:52.993945 11082 raft_consensus.cc:729] T cb7772cae3df4a99b7ee7c9c7ea1afcc P 3aaeb7323d1548299d7e4031bc1be3b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aaeb7323d1548299d7e4031bc1be3b4, State: Initialized, Role: FOLLOWER
I20260925 05:02:52.994439 11085 tablet_bootstrap.cc:492] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23: Bootstrap starting.
I20260925 05:02:52.995046 11082 consensus_queue.cc:260] T cb7772cae3df4a99b7ee7c9c7ea1afcc P 3aaeb7323d1548299d7e4031bc1be3b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aaeb7323d1548299d7e4031bc1be3b4" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36313 } } peers { permanent_uuid: "dfa792f2c0a64e309c30a8cca705f55f" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35371 } } peers { permanent_uuid: "f9a0ef2e4f8d474fa81d6d8ed5869a23" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 43427 } }
I20260925 05:02:52.996099 10929 heartbeater.cc:503] Master 127.30.247.62:41507 was elected leader, sending a full tablet report...
I20260925 05:02:52.996766 11082 ts_tablet_manager.cc:1428] T cb7772cae3df4a99b7ee7c9c7ea1afcc P 3aaeb7323d1548299d7e4031bc1be3b4: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260925 05:02:52.997814 11085 tablet_bootstrap.cc:654] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23: Neither blocks nor log segments found. Creating new log.
I20260925 05:02:52.997817 11081 raft_consensus.cc:348] T cb7772cae3df4a99b7ee7c9c7ea1afcc P dfa792f2c0a64e309c30a8cca705f55f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aaeb7323d1548299d7e4031bc1be3b4" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36313 } } peers { permanent_uuid: "dfa792f2c0a64e309c30a8cca705f55f" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35371 } } peers { permanent_uuid: "f9a0ef2e4f8d474fa81d6d8ed5869a23" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 43427 } }
I20260925 05:02:52.998184 11081 raft_consensus.cc:374] T cb7772cae3df4a99b7ee7c9c7ea1afcc P dfa792f2c0a64e309c30a8cca705f55f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:02:52.998302 11081 raft_consensus.cc:729] T cb7772cae3df4a99b7ee7c9c7ea1afcc P dfa792f2c0a64e309c30a8cca705f55f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa792f2c0a64e309c30a8cca705f55f, State: Initialized, Role: FOLLOWER
I20260925 05:02:52.998838 11081 consensus_queue.cc:260] T cb7772cae3df4a99b7ee7c9c7ea1afcc P dfa792f2c0a64e309c30a8cca705f55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aaeb7323d1548299d7e4031bc1be3b4" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36313 } } peers { permanent_uuid: "dfa792f2c0a64e309c30a8cca705f55f" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35371 } } peers { permanent_uuid: "f9a0ef2e4f8d474fa81d6d8ed5869a23" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 43427 } }
I20260925 05:02:52.998953 11085 log.cc:824] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23: Log is configured to *not* fsync() on all Append() calls
I20260925 05:02:52.999786 11063 heartbeater.cc:503] Master 127.30.247.62:41507 was elected leader, sending a full tablet report...
I20260925 05:02:53.000372 11081 ts_tablet_manager.cc:1428] T cb7772cae3df4a99b7ee7c9c7ea1afcc P dfa792f2c0a64e309c30a8cca705f55f: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260925 05:02:53.002683 11085 tablet_bootstrap.cc:492] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23: No bootstrap required, opened a new log
I20260925 05:02:53.002975 11085 ts_tablet_manager.cc:1397] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260925 05:02:53.009791 11085 raft_consensus.cc:348] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aaeb7323d1548299d7e4031bc1be3b4" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36313 } } peers { permanent_uuid: "dfa792f2c0a64e309c30a8cca705f55f" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35371 } } peers { permanent_uuid: "f9a0ef2e4f8d474fa81d6d8ed5869a23" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 43427 } }
I20260925 05:02:53.010094 11085 raft_consensus.cc:374] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:02:53.010182 11085 raft_consensus.cc:729] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9a0ef2e4f8d474fa81d6d8ed5869a23, State: Initialized, Role: FOLLOWER
I20260925 05:02:53.010707 11085 consensus_queue.cc:260] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aaeb7323d1548299d7e4031bc1be3b4" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36313 } } peers { permanent_uuid: "dfa792f2c0a64e309c30a8cca705f55f" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35371 } } peers { permanent_uuid: "f9a0ef2e4f8d474fa81d6d8ed5869a23" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 43427 } }
I20260925 05:02:53.012012 11085 ts_tablet_manager.cc:1428] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
W20260925 05:02:53.093277 10930 tablet.cc:2369] T cb7772cae3df4a99b7ee7c9c7ea1afcc P 3aaeb7323d1548299d7e4031bc1be3b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:02:53.112797 11064 tablet.cc:2369] T cb7772cae3df4a99b7ee7c9c7ea1afcc P dfa792f2c0a64e309c30a8cca705f55f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:02:53.163458 10787 tablet.cc:2369] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:02:53.168514 11089 raft_consensus.cc:482] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:02:53.168825 11089 raft_consensus.cc:504] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aaeb7323d1548299d7e4031bc1be3b4" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36313 } } peers { permanent_uuid: "dfa792f2c0a64e309c30a8cca705f55f" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35371 } } peers { permanent_uuid: "f9a0ef2e4f8d474fa81d6d8ed5869a23" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 43427 } }
I20260925 05:02:53.170423 11089 leader_election.cc:288] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aaeb7323d1548299d7e4031bc1be3b4 (127.30.247.2:36313), dfa792f2c0a64e309c30a8cca705f55f (127.30.247.3:35371)
I20260925 05:02:53.172015 11086 raft_consensus.cc:482] T cb7772cae3df4a99b7ee7c9c7ea1afcc P 3aaeb7323d1548299d7e4031bc1be3b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:02:53.172334 11086 raft_consensus.cc:504] T cb7772cae3df4a99b7ee7c9c7ea1afcc P 3aaeb7323d1548299d7e4031bc1be3b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aaeb7323d1548299d7e4031bc1be3b4" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36313 } } peers { permanent_uuid: "dfa792f2c0a64e309c30a8cca705f55f" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35371 } } peers { permanent_uuid: "f9a0ef2e4f8d474fa81d6d8ed5869a23" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 43427 } }
I20260925 05:02:53.174208 11086 leader_election.cc:288] T cb7772cae3df4a99b7ee7c9c7ea1afcc P 3aaeb7323d1548299d7e4031bc1be3b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa792f2c0a64e309c30a8cca705f55f (127.30.247.3:35371), f9a0ef2e4f8d474fa81d6d8ed5869a23 (127.30.247.1:43427)
I20260925 05:02:53.179469 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:53.170892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:38519 (local address 127.30.247.2:36313)
0925 05:02:53.171106 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:53.171113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:53.171857 (+ 744us) server_negotiation.cc:407] Connection header received
0925 05:02:53.172166 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:53.172173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:53.172254 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:53.172354 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:02:53.173206 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:53.174271 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:53.177019 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:53.178306 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:53.178425 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:53.178494 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:53.179056 (+ 562us) server_negotiation.cc:299] Negotiation successful
0925 05:02:53.179178 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:02:53.180209 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:53.170909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:36313 (local address 127.30.247.1:38519)
0925 05:02:53.171724 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0925 05:02:53.171748 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 05:02:53.171960 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:02:53.172437 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:02:53.172441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:02:53.172468 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:02:53.172985 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:53.172994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:53.174427 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:53.174431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:53.176844 (+ 2413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:53.176858 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:53.178687 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:53.178693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:53.178860 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:53.178898 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 05:02:53.179690 (+ 792us) client_negotiation.cc:240] Negotiation successful
0925 05:02:53.179814 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":79,"spinlock_wait_cycles":44928,"thread_start_us":440,"threads_started":1}
I20260925 05:02:53.180344 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:53.170947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:50855 (local address 127.30.247.3:35371)
0925 05:02:53.171153 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:53.171160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:53.171907 (+ 747us) server_negotiation.cc:407] Connection header received
0925 05:02:53.172081 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:53.172087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:53.172167 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:53.172251 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:02:53.173096 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:53.174089 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:53.177700 (+ 3611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:53.178983 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:53.179096 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:53.179170 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:53.179947 (+ 777us) server_negotiation.cc:299] Negotiation successful
0925 05:02:53.180070 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:02:53.181085 10884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7772cae3df4a99b7ee7c9c7ea1afcc" candidate_uuid: "f9a0ef2e4f8d474fa81d6d8ed5869a23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aaeb7323d1548299d7e4031bc1be3b4" is_pre_election: true
I20260925 05:02:53.181579 10884 raft_consensus.cc:2454] T cb7772cae3df4a99b7ee7c9c7ea1afcc P 3aaeb7323d1548299d7e4031bc1be3b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9a0ef2e4f8d474fa81d6d8ed5869a23 in term 0.
I20260925 05:02:53.181965 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:53.170960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:35371 (local address 127.30.247.1:50855)
0925 05:02:53.171743 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0925 05:02:53.171757 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:02:53.171927 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:02:53.172343 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:02:53.172349 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:02:53.172378 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:02:53.172903 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:53.172919 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:53.174557 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:53.174563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:53.177534 (+ 2971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:53.177547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:53.179346 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:53.179362 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:53.179767 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:53.179800 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 05:02:53.181440 (+ 1640us) client_negotiation.cc:240] Negotiation successful
0925 05:02:53.181575 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":30,"thread_start_us":484,"threads_started":1}
I20260925 05:02:53.182307 10674 leader_election.cc:302] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aaeb7323d1548299d7e4031bc1be3b4, f9a0ef2e4f8d474fa81d6d8ed5869a23; no voters:
I20260925 05:02:53.183133 11018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7772cae3df4a99b7ee7c9c7ea1afcc" candidate_uuid: "f9a0ef2e4f8d474fa81d6d8ed5869a23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa792f2c0a64e309c30a8cca705f55f" is_pre_election: true
I20260925 05:02:53.183691 11018 raft_consensus.cc:2454] T cb7772cae3df4a99b7ee7c9c7ea1afcc P dfa792f2c0a64e309c30a8cca705f55f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9a0ef2e4f8d474fa81d6d8ed5869a23 in term 0.
I20260925 05:02:53.184608 11089 raft_consensus.cc:2781] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:02:53.184751 11089 raft_consensus.cc:482] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:02:53.184840 11089 raft_consensus.cc:3037] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:02:53.188079 11089 raft_consensus.cc:504] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aaeb7323d1548299d7e4031bc1be3b4" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36313 } } peers { permanent_uuid: "dfa792f2c0a64e309c30a8cca705f55f" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35371 } } peers { permanent_uuid: "f9a0ef2e4f8d474fa81d6d8ed5869a23" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 43427 } }
I20260925 05:02:53.189260 10884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7772cae3df4a99b7ee7c9c7ea1afcc" candidate_uuid: "f9a0ef2e4f8d474fa81d6d8ed5869a23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aaeb7323d1548299d7e4031bc1be3b4"
I20260925 05:02:53.189261 11018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7772cae3df4a99b7ee7c9c7ea1afcc" candidate_uuid: "f9a0ef2e4f8d474fa81d6d8ed5869a23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa792f2c0a64e309c30a8cca705f55f"
I20260925 05:02:53.189538 11089 leader_election.cc:288] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23 [CANDIDATE]: Term 1 election: Requested vote from peers 3aaeb7323d1548299d7e4031bc1be3b4 (127.30.247.2:36313), dfa792f2c0a64e309c30a8cca705f55f (127.30.247.3:35371)
I20260925 05:02:53.189541 11018 raft_consensus.cc:3037] T cb7772cae3df4a99b7ee7c9c7ea1afcc P dfa792f2c0a64e309c30a8cca705f55f [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:02:53.189605 10884 raft_consensus.cc:3037] T cb7772cae3df4a99b7ee7c9c7ea1afcc P 3aaeb7323d1548299d7e4031bc1be3b4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:02:53.190223 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:53.174732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:41101 (local address 127.30.247.3:35371)
0925 05:02:53.180860 (+ 6128us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:53.180865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:53.180881 (+ 16us) server_negotiation.cc:407] Connection header received
0925 05:02:53.180931 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:53.180935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:53.181004 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:53.181089 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:02:53.183314 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:53.185047 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:53.187721 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:53.189018 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:53.189171 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:53.189235 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:53.189814 (+ 579us) server_negotiation.cc:299] Negotiation successful
0925 05:02:53.189944 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6031,"thread_start_us":12198,"threads_started":1}
I20260925 05:02:53.190958 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:53.174619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:35371 (local address 127.30.247.2:41101)
0925 05:02:53.175954 (+ 1335us) negotiation.cc:107] Waiting for socket to connect
0925 05:02:53.175976 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:02:53.176081 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:02:53.181247 (+ 5166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:02:53.181257 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:02:53.181281 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:02:53.183138 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:53.183152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:53.185355 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:53.185359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:53.187566 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:53.187580 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:53.189389 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:53.189395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:53.189596 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:53.189631 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 05:02:53.190419 (+ 788us) client_negotiation.cc:240] Negotiation successful
0925 05:02:53.190557 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1215,"thread_start_us":1040,"threads_started":1}
I20260925 05:02:53.191725 11017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7772cae3df4a99b7ee7c9c7ea1afcc" candidate_uuid: "3aaeb7323d1548299d7e4031bc1be3b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa792f2c0a64e309c30a8cca705f55f" is_pre_election: true
I20260925 05:02:53.192647 11018 raft_consensus.cc:2454] T cb7772cae3df4a99b7ee7c9c7ea1afcc P dfa792f2c0a64e309c30a8cca705f55f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9a0ef2e4f8d474fa81d6d8ed5869a23 in term 1.
I20260925 05:02:53.192937 10884 raft_consensus.cc:2454] T cb7772cae3df4a99b7ee7c9c7ea1afcc P 3aaeb7323d1548299d7e4031bc1be3b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9a0ef2e4f8d474fa81d6d8ed5869a23 in term 1.
I20260925 05:02:53.192999 10675 leader_election.cc:302] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dfa792f2c0a64e309c30a8cca705f55f, f9a0ef2e4f8d474fa81d6d8ed5869a23; no voters:
I20260925 05:02:53.193293 11089 raft_consensus.cc:2781] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:02:53.193819 11089 raft_consensus.cc:686] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23 [term 1 LEADER]: Becoming Leader. State: Replica: f9a0ef2e4f8d474fa81d6d8ed5869a23, State: Running, Role: LEADER
I20260925 05:02:53.194347 11089 consensus_queue.cc:237] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aaeb7323d1548299d7e4031bc1be3b4" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36313 } } peers { permanent_uuid: "dfa792f2c0a64e309c30a8cca705f55f" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35371 } } peers { permanent_uuid: "f9a0ef2e4f8d474fa81d6d8ed5869a23" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 43427 } }
I20260925 05:02:53.195554 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:53.176583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:42211 (local address 127.30.247.1:43427)
0925 05:02:53.176750 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:53.176757 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:53.187189 (+ 10432us) server_negotiation.cc:407] Connection header received
0925 05:02:53.187400 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:53.187405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:53.187467 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:53.187510 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:02:53.188197 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:53.189270 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:53.193696 (+ 4426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:53.194670 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:53.194842 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:53.194909 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:53.195207 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 05:02:53.195332 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 05:02:53.195602 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:53.176162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:43427 (local address 127.30.247.2:42211)
0925 05:02:53.187070 (+ 10908us) negotiation.cc:107] Waiting for socket to connect
0925 05:02:53.187087 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:02:53.187241 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:02:53.187682 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:02:53.187686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:02:53.187712 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:02:53.188031 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:53.188041 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:53.190578 (+ 2537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:53.190595 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:53.193558 (+ 2963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:02:53.193568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:02:53.194866 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:02:53.194871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:02:53.195054 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:53.195090 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 05:02:53.195180 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 05:02:53.195293 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10786,"thread_start_us":11033,"threads_started":1}
I20260925 05:02:53.196456 10741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7772cae3df4a99b7ee7c9c7ea1afcc" candidate_uuid: "3aaeb7323d1548299d7e4031bc1be3b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a0ef2e4f8d474fa81d6d8ed5869a23" is_pre_election: true
I20260925 05:02:53.197445 10818 leader_election.cc:302] T cb7772cae3df4a99b7ee7c9c7ea1afcc P 3aaeb7323d1548299d7e4031bc1be3b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aaeb7323d1548299d7e4031bc1be3b4; no voters: dfa792f2c0a64e309c30a8cca705f55f, f9a0ef2e4f8d474fa81d6d8ed5869a23
I20260925 05:02:53.197791 11086 raft_consensus.cc:2726] T cb7772cae3df4a99b7ee7c9c7ea1afcc P 3aaeb7323d1548299d7e4031bc1be3b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 05:02:53.199222 10572 catalog_manager.cc:5495] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9a0ef2e4f8d474fa81d6d8ed5869a23 (127.30.247.1). New cstate: current_term: 1 leader_uuid: "f9a0ef2e4f8d474fa81d6d8ed5869a23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aaeb7323d1548299d7e4031bc1be3b4" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa792f2c0a64e309c30a8cca705f55f" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9a0ef2e4f8d474fa81d6d8ed5869a23" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 43427 } health_report { overall_health: HEALTHY } } }
I20260925 05:02:53.243934 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:53.238678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52488 (local address 127.30.247.1:43427)
0925 05:02:53.238845 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:53.238850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:53.239046 (+ 196us) server_negotiation.cc:407] Connection header received
0925 05:02:53.239204 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:53.239208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:53.239300 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:53.239427 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:02:53.239775 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:53.240469 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:53.242002 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:53.242403 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:53.242502 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:53.242947 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:02:53.243449 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:02:53.243553 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:53.243649 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 05:02:53.243709 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 05:02:53.251940 10884 raft_consensus.cc:1261] T cb7772cae3df4a99b7ee7c9c7ea1afcc P 3aaeb7323d1548299d7e4031bc1be3b4 [term 1 FOLLOWER]: Refusing update from remote peer f9a0ef2e4f8d474fa81d6d8ed5869a23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:02:53.251974 11018 raft_consensus.cc:1261] T cb7772cae3df4a99b7ee7c9c7ea1afcc P dfa792f2c0a64e309c30a8cca705f55f [term 1 FOLLOWER]: Refusing update from remote peer f9a0ef2e4f8d474fa81d6d8ed5869a23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:02:53.252808 11095 consensus_queue.cc:1035] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aaeb7323d1548299d7e4031bc1be3b4" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:02:53.253141 11089 consensus_queue.cc:1035] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa792f2c0a64e309c30a8cca705f55f" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:02:53.263652 11098 mvcc.cc:205] Tried to move back new op lower bound from 7333120300030664704 to 7333120299807744000. Current Snapshot: MvccSnapshot[applied={T|T < 7333120300030664704}]
I20260925 05:02:53.267009 11099 mvcc.cc:205] Tried to move back new op lower bound from 7333120300030664704 to 7333120299807744000. Current Snapshot: MvccSnapshot[applied={T|T < 7333120300030664704}]
I20260925 05:02:53.267007 11100 mvcc.cc:205] Tried to move back new op lower bound from 7333120300030664704 to 7333120299807744000. Current Snapshot: MvccSnapshot[applied={T|T < 7333120300030664704}]
I20260925 05:02:53.279115 10571 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:02:53.279477 10571 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:02:53.284413 10571 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e34710f690142e0b1320218be91924a: Sending DeleteTablet for 3 replicas of tablet cb7772cae3df4a99b7ee7c9c7ea1afcc
I20260925 05:02:53.285689 10998 tablet_service.cc:1514] Processing DeleteTablet for tablet cb7772cae3df4a99b7ee7c9c7ea1afcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:02:53 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:43378
I20260925 05:02:53.286017 10864 tablet_service.cc:1514] Processing DeleteTablet for tablet cb7772cae3df4a99b7ee7c9c7ea1afcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:02:53 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:32902
I20260925 05:02:53.286139 10721 tablet_service.cc:1514] Processing DeleteTablet for tablet cb7772cae3df4a99b7ee7c9c7ea1afcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:02:53 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:52472
I20260925 05:02:53.286749 10720 tablet_service.cc:1514] Processing DeleteTablet for tablet cb7772cae3df4a99b7ee7c9c7ea1afcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:02:53 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:52472
I20260925 05:02:53.289058 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 10653
I20260925 05:02:53.315678 11109 tablet_replica.cc:330] T cb7772cae3df4a99b7ee7c9c7ea1afcc P 3aaeb7323d1548299d7e4031bc1be3b4: stopping tablet replica
I20260925 05:02:53.316300 11109 raft_consensus.cc:2229] T cb7772cae3df4a99b7ee7c9c7ea1afcc P 3aaeb7323d1548299d7e4031bc1be3b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:02:53.319178 11109 raft_consensus.cc:2258] T cb7772cae3df4a99b7ee7c9c7ea1afcc P 3aaeb7323d1548299d7e4031bc1be3b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:02:53.321177 11110 tablet_replica.cc:330] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23: stopping tablet replica
I20260925 05:02:53.321704 11110 raft_consensus.cc:2229] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:02:53.322525 11110 raft_consensus.cc:2258] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:02:53.325199 11110 ts_tablet_manager.cc:1905] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:02:53.330356 11109 ts_tablet_manager.cc:1905] T cb7772cae3df4a99b7ee7c9c7ea1afcc P 3aaeb7323d1548299d7e4031bc1be3b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:02:53.332724 11110 ts_tablet_manager.cc:1918] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:02:53.332948 11110 log.cc:1196] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-0/wal/wals/cb7772cae3df4a99b7ee7c9c7ea1afcc
I20260925 05:02:53.332994 11108 tablet_replica.cc:330] T cb7772cae3df4a99b7ee7c9c7ea1afcc P dfa792f2c0a64e309c30a8cca705f55f: stopping tablet replica
I20260925 05:02:53.333362 11108 raft_consensus.cc:2229] T cb7772cae3df4a99b7ee7c9c7ea1afcc P dfa792f2c0a64e309c30a8cca705f55f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:02:53.333518 11110 ts_tablet_manager.cc:1939] T cb7772cae3df4a99b7ee7c9c7ea1afcc P f9a0ef2e4f8d474fa81d6d8ed5869a23: Deleting consensus metadata
I20260925 05:02:53.333823 11108 raft_consensus.cc:2258] T cb7772cae3df4a99b7ee7c9c7ea1afcc P dfa792f2c0a64e309c30a8cca705f55f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:02:53.335253 10557 catalog_manager.cc:4841] TS f9a0ef2e4f8d474fa81d6d8ed5869a23 (127.30.247.1:43427): tablet cb7772cae3df4a99b7ee7c9c7ea1afcc (table test-table [id=4c06e58583c242b9b0fbae927835a375]) successfully deleted
W20260925 05:02:53.336265 10557 catalog_manager.cc:4808] TS f9a0ef2e4f8d474fa81d6d8ed5869a23 (127.30.247.1:43427): delete failed for tablet cb7772cae3df4a99b7ee7c9c7ea1afcc because the tablet was not found. No further retry: Not found: Tablet not found: cb7772cae3df4a99b7ee7c9c7ea1afcc
I20260925 05:02:53.336786 11108 ts_tablet_manager.cc:1905] T cb7772cae3df4a99b7ee7c9c7ea1afcc P dfa792f2c0a64e309c30a8cca705f55f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:02:53.341988 11108 ts_tablet_manager.cc:1918] T cb7772cae3df4a99b7ee7c9c7ea1afcc P dfa792f2c0a64e309c30a8cca705f55f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:02:53.342144 11108 log.cc:1196] T cb7772cae3df4a99b7ee7c9c7ea1afcc P dfa792f2c0a64e309c30a8cca705f55f: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-2/wal/wals/cb7772cae3df4a99b7ee7c9c7ea1afcc
I20260925 05:02:53.342628 11108 ts_tablet_manager.cc:1939] T cb7772cae3df4a99b7ee7c9c7ea1afcc P dfa792f2c0a64e309c30a8cca705f55f: Deleting consensus metadata
Sep 25 05:02:53 dist-test-slave-5n86 krb5kdc[10096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.1@KRBTEST.COM
I20260925 05:02:53.344237 10557 catalog_manager.cc:4841] TS dfa792f2c0a64e309c30a8cca705f55f (127.30.247.3:35371): tablet cb7772cae3df4a99b7ee7c9c7ea1afcc (table test-table [id=4c06e58583c242b9b0fbae927835a375]) successfully deleted
I20260925 05:02:53.350066 11109 ts_tablet_manager.cc:1918] T cb7772cae3df4a99b7ee7c9c7ea1afcc P 3aaeb7323d1548299d7e4031bc1be3b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:02:53.350296 11109 log.cc:1196] T cb7772cae3df4a99b7ee7c9c7ea1afcc P 3aaeb7323d1548299d7e4031bc1be3b4: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312430481417-31708-0/minicluster-data/ts-1/wal/wals/cb7772cae3df4a99b7ee7c9c7ea1afcc
I20260925 05:02:53.350790 11109 ts_tablet_manager.cc:1939] T cb7772cae3df4a99b7ee7c9c7ea1afcc P 3aaeb7323d1548299d7e4031bc1be3b4: Deleting consensus metadata
I20260925 05:02:53.352524 10556 catalog_manager.cc:4841] TS 3aaeb7323d1548299d7e4031bc1be3b4 (127.30.247.2:36313): tablet cb7772cae3df4a99b7ee7c9c7ea1afcc (table test-table [id=4c06e58583c242b9b0fbae927835a375]) successfully deleted
I20260925 05:02:53.352784 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:53.290524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52494 (local address 127.30.247.1:43427)
0925 05:02:53.290763 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:53.290773 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:53.314799 (+ 24026us) server_negotiation.cc:407] Connection header received
0925 05:02:53.316962 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:53.316972 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:53.317087 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:53.317185 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:53.324711 (+ 7526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:53.326205 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:53.334213 (+ 8008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:53.334701 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:53.334782 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:53.345157 (+ 10375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:02:53.345171 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:02:53.345186 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:02:53.345236 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:02:53.349282 (+ 4046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:53.350195 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:53.350205 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:53.350209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:53.350363 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:53.350818 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:53.350824 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:53.350827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:53.351319 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:02:53.351615 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:53.352183 (+ 568us) server_negotiation.cc:299] Negotiation successful
0925 05:02:53.352424 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 05:02:53.354230 10781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:02:53.455715 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 10653
I20260925 05:02:53.473053 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 10792
Sep 25 05:02:53 dist-test-slave-5n86 krb5kdc[10096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.2@KRBTEST.COM
I20260925 05:02:53.487425 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:53.473881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32904 (local address 127.30.247.2:36313)
0925 05:02:53.474073 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:53.474077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:53.474095 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:02:53.474148 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:53.474152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:53.474212 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:53.474304 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:53.475522 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:53.476257 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:53.477529 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:53.477919 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:53.478012 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:53.481955 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:02:53.481966 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:02:53.481980 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:02:53.482014 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:02:53.484753 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:53.485387 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:53.485394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:53.485396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:53.485520 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:53.485894 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:53.485897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:53.485899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:53.486302 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:02:53.486514 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:53.486817 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 05:02:53.487073 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 05:02:53.488050 10924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:02:53.578588 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 10792
I20260925 05:02:53.594769 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 10935
Sep 25 05:02:53 dist-test-slave-5n86 krb5kdc[10096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.3@KRBTEST.COM
I20260925 05:02:53.609297 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:02:53.595531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43394 (local address 127.30.247.3:35371)
0925 05:02:53.595721 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 05:02:53.595725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:02:53.595743 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:02:53.595883 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:02:53.595887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:02:53.595963 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:02:53.596054 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:02:53.597429 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:53.598174 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:53.599447 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:02:53.599851 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:02:53.599945 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:02:53.604157 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:02:53.604173 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:02:53.604190 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:02:53.604219 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:02:53.606917 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:53.607516 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:53.607523 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:53.607525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:53.607639 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:02:53.607941 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:02:53.607944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:02:53.607946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:02:53.608238 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:02:53.608457 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 05:02:53.608813 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 05:02:53.608984 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 05:02:53.609951 11058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:02:53.697319 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 10935
I20260925 05:02:53.713229 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 10540
I20260925 05:02:53.714350 10602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:02:53.826138 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 10540
I20260925 05:02:53.847838 31708 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 05:02:54.182317 31708 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 05:02:54.182554 31708 mini_ranger.cc:66] Stopping Ranger...
I20260925 05:02:54.527336 31708 mini_ranger.cc:68] Stopped Ranger
2026-09-25 05:02:54.527 UTC [10126] LOG: received smart shutdown request
2026-09-25 05:02:54.531 UTC [10126] LOG: background worker "logical replication launcher" (PID 10135) exited with exit code 1
2026-09-25 05:02:54.531 UTC [10130] LOG: shutting down
2026-09-25 05:02:54.722 UTC [10126] LOG: database system is shut down
2026-09-25T05:02:54Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47985 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:02:54 dist-test-slave-5n86 krb5kdc[11124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:02:54 dist-test-slave-5n86 krb5kdc[11124](info): set up 2 sockets
Sep 25 05:02:54 dist-test-slave-5n86 krb5kdc[11124](info): commencing operation
krb5kdc: starting...
W20260925 05:02:55.949103 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:02:56 dist-test-slave-5n86 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:02:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:02:56Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.30.247.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.30.247.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.30.247.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/rangeradmin_127.30.247.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.30.247.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/rangeradmin_127.30.247.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.30.247.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.30.247.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.30.247.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/rangerlookup_127.30.247.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.30.247.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/rangerlookup_127.30.247.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.247.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.30.247.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.247.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/HTTP_127.30.247.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.247.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/HTTP_127.30.247.20@KRBTEST.COM.keytab.
I20260925 05:02:56.251086 31708 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task43xS0j/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/postgres -l logfile start
2026-09-25 05:02:58.873 UTC [11154] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 05:02:58.873 UTC [11154] LOG: listening on IPv4 address "127.30.247.20", port 54537
2026-09-25 05:02:58.877 UTC [11154] LOG: listening on Unix socket "/tmp/.s.PGSQL.54537"
2026-09-25 05:02:58.901 UTC [11157] LOG: database system was shut down at 2026-09-25 05:02:57 UTC
2026-09-25 05:02:58.904 UTC [11154] LOG: database system is ready to accept connections
I20260925 05:02:59.868880 31708 mini_postgres.cc:96] Postgres bound to 54537
2026-09-25 05:02:59.890 UTC [11165] FATAL: database "slave" does not exist
127.30.247.20:54537 - accepting connections
I20260925 05:02:59.891259 31708 mini_ranger.cc:161] Starting Ranger...
I20260925 05:02:59.916043 31708 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 05:03:00.435529 31708 mini_ranger.cc:87] Created ranger Postgres database
I20260925 05:03:00.435654 31708 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ranger-admin
2026-09-25 05:03:00,696 [I] DB FLAVOR :POSTGRES
2026-09-25 05:03:00,697 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 05:03:00,697 [I] Checking connection..
2026-09-25 05:03:00,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:54537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 05:03:00,944 [I] Checking connection passed.
2026-09-25 05:03:00,944 [I] --------- Verifying version history table ---------
2026-09-25 05:03:00,944 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:54537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 05:03:01,183 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 05:03:01,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:54537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 05:03:01,436 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 05:03:01,436 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 05:03:01,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:54537/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 05:03:01.689 UTC [11229] WARNING: there is no transaction in progress
2026-09-25 05:03:01,695 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 05:03:01,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:54537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 05:03:01,936 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 05:03:01,936 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 05:03:01,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:54537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 05:03:02,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:54537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 05:03:02,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:54537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5n86.c.gcp-upstream.internal','N') ;"
2026-09-25 05:03:02,678 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:54537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 05:03:02,934 [I] Table x_portal_user does not exist in database ranger
2026-09-25 05:03:02,934 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:54537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 05:03:03,187 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 05:03:03,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:54537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 05:03:03,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:54537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 05:03:03,666 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 05:03:03,666 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:54537/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 05:03:04.638 UTC [11337] WARNING: there is no transaction in progress
2026-09-25 05:03:04.654 UTC [11337] WARNING: there is no transaction in progress
2026-09-25 05:03:04.671 UTC [11337] WARNING: there is no transaction in progress
2026-09-25 05:03:04.686 UTC [11337] WARNING: there is no transaction in progress
2026-09-25 05:03:04.705 UTC [11337] WARNING: there is no transaction in progress
2026-09-25 05:03:04.723 UTC [11337] WARNING: there is no transaction in progress
2026-09-25 05:03:04.776 UTC [11337] WARNING: there is no transaction in progress
2026-09-25 05:03:04.784 UTC [11337] WARNING: there is no transaction in progress
2026-09-25 05:03:04.798 UTC [11337] WARNING: there is no transaction in progress
2026-09-25 05:03:04.808 UTC [11337] WARNING: there is no transaction in progress
2026-09-25 05:03:04.819 UTC [11337] WARNING: there is no transaction in progress
2026-09-25 05:03:04.832 UTC [11337] WARNING: there is no transaction in progress
2026-09-25 05:03:04.842 UTC [11337] WARNING: there is no transaction in progress
2026-09-25 05:03:04.849 UTC [11337] WARNING: there is no transaction in progress
2026-09-25 05:03:04.856 UTC [11337] WARNING: there is no transaction in progress
2026-09-25 05:03:04.864 UTC [11337] WARNING: there is no transaction in progress
2026-09-25 05:03:05,544 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 05:03:05,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:54537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 05:03:05,794 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 05:03:05,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:54537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 05:03:06,048 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 05:03:06,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:54537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 05:03:06,296 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 05:03:06,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:54537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 05:03:06,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:54537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 05:03:06,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:54537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5n86.c.gcp-upstream.internal';"
2026-09-25 05:03:07,075 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 05:03:07,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:54537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 05:03:07,295 [I] DB_PATCHES have already been applied
I20260925 05:03:07.301767 31708 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ranger-admin:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task43xS0j/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 05:03:07.301885 31708 mini_ranger.cc:193] Using host: 127.30.247.20
I20260925 05:03:07.315918 31708 mini_ranger.cc:227] Ranger admin URL: http://127.30.247.20:46997
Sep 25, 2026 5:03:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 5:03:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 5:03:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 5:03:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 5:03:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 5:03:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 5:03:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 05:03:08.475015 31708 mini_ranger.cc:160] Time spent starting Ranger: real 8.584s user 0.000s sys 0.035s
Sep 25, 2026 5:03:28 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 5:03:28 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 5:03:28 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 5:03:28 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 5:03:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 5:03:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 5:03:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 5:03:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 5:03:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 5:03:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 5:03:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 5:03:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 5:03:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 5:03:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 5:03:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 5:03:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 5:03:28 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 5:03:29 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 05:03:29.784947 31708 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.30.247.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.30.247.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.30.247.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/rangerkms_127.30.247.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.30.247.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/rangerkms_127.30.247.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.247.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.30.247.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.247.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/HTTP_127.30.247.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.247.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/HTTP_127.30.247.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 05:03:29 dist-test-slave-5n86 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312609, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 05:03:29.948709 31708 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 05:03:29.971771 31708 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 05:03:30.930321 31708 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 05:03:30.930420 31708 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ranger-kms
I20260925 05:03:30.930471 31708 mini_ranger_kms.cc:227] Using postgres at 127.30.247.20:54537
2026-09-25 05:03:31,273 [I] DB FLAVOR :POSTGRES
2026-09-25 05:03:31,273 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 05:03:31,273 [I] Checking connection
2026-09-25 05:03:31,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:54537/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 05:03:31,523 [I] connection success
2026-09-25 05:03:31,524 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 05:03:31,524 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 05:03:31,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:54537/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 05:03:31,746 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 05:03:31,746 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 05:03:31,747 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 05:03:31,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.247.20:54537/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 05:03:32,010 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 05:03:32.486505 31708 mini_ranger_kms.cc:310] Created kms service
I20260925 05:03:32.693096 31708 mini_ranger_kms.cc:326] Created kudu user
I20260925 05:03:32.737421 31708 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 05:03:32 dist-test-slave-5n86 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312609, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.30.247.20@KRBTEST.COM
I20260925 05:03:33.113117 31708 mini_ranger_kms.cc:384] Added ranger policy
I20260925 05:03:33.113430 31708 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ranger-kms:/tmp/dist-test-task43xS0j/build/asan/bin/postgresql.jar:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task43xS0j/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 05:03:33.113478 31708 mini_ranger_kms.cc:240] Using host: 127.30.247.20
I20260925 05:03:33.131100 31708 mini_ranger_kms.cc:276] Ranger KMS PID: 11517
I20260925 05:03:33.131238 31708 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.30.247.20:46799
Sep 25, 2026 5:03:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 5:03:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 5:03:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 5:03:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 5:03:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 5:03:34 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 05:03:35.359217 31708 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.411s user 0.099s sys 0.254s
I20260925 05:03:35.359473 31708 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 05:03:35.359655 31708 mini_ranger_kms.cc:401] 127.30.247.20:46799/kms/v1/keys
Sep 25 05:03:35 dist-test-slave-5n86 krb5kdc[11124](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790312615, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.247.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 05:03:35 dist-test-slave-5n86 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312615, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.247.20@KRBTEST.COM for HTTP/127.30.247.20@KRBTEST.COM
Sep 25 05:03:36 dist-test-slave-5n86 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312615, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.247.20@KRBTEST.COM for HTTP/127.30.247.20@KRBTEST.COM
Sep 25 05:03:38 dist-test-slave-5n86 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:03:38.433485 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:46641
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:46641
--ranger_config_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.247.20:46799/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:03:38.758095 11568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:03:38.758361 11568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:03:38.758448 11568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:03:38.765465 11568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:03:38.765558 11568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:03:38.765602 11568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:03:38.765658 11568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:03:38.765699 11568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:03:38.775157 11568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/wal
--ranger_kms_url=127.30.247.20:46799/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:46641
--ranger_config_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:46641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:03:38.776314 11568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:03:38.777575 11568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:03:38.788878 11573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:38.788921 11576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:38.788946 11574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:03:38.789705 11568 server_base.cc:1034] running on GCE node
I20260925 05:03:38.790352 11568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:03:38.791621 11568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:03:38.792807 11568 hybrid_clock.cc:648] HybridClock initialized: now 1790312618792767 us; error 33 us; skew 500 ppm
I20260925 05:03:38.793081 11568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:03:38 dist-test-slave-5n86 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312618, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:03:38.798945 11568 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:03:38.800321 11568 webserver.cc:460] Webserver started at http://127.30.247.62:33929/ using document root <none> and password file <none>
I20260925 05:03:38.800944 11568 fs_manager.cc:359] Metadata directory not provided
I20260925 05:03:38.801038 11568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:03:38.801281 11568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:03:38 dist-test-slave-5n86 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312618, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for HTTP/127.30.247.20@KRBTEST.COM
I20260925 05:03:38.861857 11568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "a25c638006f44a2eafeb5c8625f882d4"
format_stamp: "Formatted at 2026-09-25 05:03:38 on dist-test-slave-5n86"
server_key: "b13f5a5e4da2f2fa4b0fabe291023cc4"
server_key_iv: "7e703214b0cc72653796f61307987753"
server_key_version: "kuduclusterkey@0"
I20260925 05:03:38.862789 11568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "a25c638006f44a2eafeb5c8625f882d4"
format_stamp: "Formatted at 2026-09-25 05:03:38 on dist-test-slave-5n86"
server_key: "b13f5a5e4da2f2fa4b0fabe291023cc4"
server_key_iv: "7e703214b0cc72653796f61307987753"
server_key_version: "kuduclusterkey@0"
I20260925 05:03:38.868083 11568 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 05:03:38.884584 11583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:38.886129 11568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:03:38.886312 11568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "a25c638006f44a2eafeb5c8625f882d4"
format_stamp: "Formatted at 2026-09-25 05:03:38 on dist-test-slave-5n86"
server_key: "b13f5a5e4da2f2fa4b0fabe291023cc4"
server_key_iv: "7e703214b0cc72653796f61307987753"
server_key_version: "kuduclusterkey@0"
I20260925 05:03:38.886544 11568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:03:38.903455 11568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:03:38.907214 11568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:03:38.907505 11568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:03:38.926329 11568 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:46641
I20260925 05:03:38.926329 11634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:46641 every 8 connection(s)
I20260925 05:03:38.927596 11568 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:03:38.928586 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 11568
I20260925 05:03:38.928848 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:03:38.929889 31708 external_mini_cluster.cc:1428] Setting key 9b1570746788d8d0612581c8bb2816ee
I20260925 05:03:38.932963 11635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 05:03:38 dist-test-slave-5n86 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:03:38.948434 11635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4: Bootstrap starting.
I20260925 05:03:38.952329 11635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4: Neither blocks nor log segments found. Creating new log.
I20260925 05:03:38.953860 11635 log.cc:824] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4: Log is configured to *not* fsync() on all Append() calls
I20260925 05:03:38.954123 11638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:38.932196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37646 (local address 127.30.247.62:46641)
0925 05:03:38.936019 (+ 3823us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:38.936030 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:38.936074 (+ 44us) server_negotiation.cc:407] Connection header received
0925 05:03:38.938031 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:38.938052 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:38.938414 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:38.938800 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:38.940158 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:38.941507 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:38.942780 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:38.943419 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:38.943610 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:38.947324 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:38.947332 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:38.947345 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:38.947383 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:38.950344 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:38.951029 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:38.951037 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:38.951052 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:38.951203 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:38.951592 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:38.951596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:38.951598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:38.952062 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:38.952379 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:38.952778 (+ 399us) server_negotiation.cc:299] Negotiation successful
0925 05:03:38.953155 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3580,"thread_start_us":3343,"threads_started":1}
I20260925 05:03:38.957300 11635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4: No bootstrap required, opened a new log
I20260925 05:03:38.967849 11635 raft_consensus.cc:348] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a25c638006f44a2eafeb5c8625f882d4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46641 } }
I20260925 05:03:38.968261 11635 raft_consensus.cc:374] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:03:38.968361 11635 raft_consensus.cc:729] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a25c638006f44a2eafeb5c8625f882d4, State: Initialized, Role: FOLLOWER
I20260925 05:03:38.968973 11635 consensus_queue.cc:260] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a25c638006f44a2eafeb5c8625f882d4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46641 } }
I20260925 05:03:38.969236 11635 raft_consensus.cc:388] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:03:38.969368 11635 raft_consensus.cc:482] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:03:38.969524 11635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:03:38.972813 11635 raft_consensus.cc:504] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a25c638006f44a2eafeb5c8625f882d4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46641 } }
I20260925 05:03:38.973358 11635 leader_election.cc:302] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a25c638006f44a2eafeb5c8625f882d4; no voters:
I20260925 05:03:38.973990 11635 leader_election.cc:288] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:03:38.974099 11640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:03:38.975538 11640 raft_consensus.cc:686] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [term 1 LEADER]: Becoming Leader. State: Replica: a25c638006f44a2eafeb5c8625f882d4, State: Running, Role: LEADER
I20260925 05:03:38.976027 11635 sys_catalog.cc:564] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:03:38.976065 11640 consensus_queue.cc:237] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a25c638006f44a2eafeb5c8625f882d4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46641 } }
I20260925 05:03:38.978693 11642 sys_catalog.cc:455] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a25c638006f44a2eafeb5c8625f882d4. Latest consensus state: current_term: 1 leader_uuid: "a25c638006f44a2eafeb5c8625f882d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a25c638006f44a2eafeb5c8625f882d4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46641 } } }
I20260925 05:03:38.979048 11642 sys_catalog.cc:458] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [sys.catalog]: This master's current role is: LEADER
I20260925 05:03:38.979528 11641 sys_catalog.cc:455] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a25c638006f44a2eafeb5c8625f882d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a25c638006f44a2eafeb5c8625f882d4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46641 } } }
I20260925 05:03:38.979807 11641 sys_catalog.cc:458] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [sys.catalog]: This master's current role is: LEADER
I20260925 05:03:38.981815 11645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:03:38.986967 11645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:03:38.997666 11635 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 05:03:38.999213 11645 catalog_manager.cc:1347] Generated new cluster ID: 7f46ba2ee4e6438bbfae9eb69da74b6b
I20260925 05:03:38.999347 11645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:03:39.038272 11645 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:03:39.039652 11645 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:03:39.055784 11645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4: Generated new TSK 0
I20260925 05:03:39.056926 11645 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
I20260925 05:03:39.742563 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.247.20:46799/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:0
--local_ip_for_outbound_sockets=127.30.247.1
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:46641
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:03:40 dist-test-slave-5n86 krb5kdc[11124](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790312620, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 05:03:40.146493 11680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:03:40.146795 11680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:03:40.146958 11680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:03:40.155161 11680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:03:40.155294 11680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:03:40.155505 11680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:03:40.166095 11680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.30.247.20:46799/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:46641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:03:40.167392 11680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:03:40.168839 11680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:03:40.186527 11690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:40.187399 11691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:40.196606 11693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:03:40.197103 11680 server_base.cc:1034] running on GCE node
I20260925 05:03:40.197849 11680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:03:40.199074 11680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:03:40.200405 11680 hybrid_clock.cc:648] HybridClock initialized: now 1790312620200294 us; error 96 us; skew 500 ppm
I20260925 05:03:40.200834 11680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:03:40 dist-test-slave-5n86 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312620, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:03:40.204851 11680 init.cc:377] Logged in from keytab as kudu/127.30.247.1@KRBTEST.COM (short username kudu)
I20260925 05:03:40.206688 11680 webserver.cc:460] Webserver started at http://127.30.247.1:34201/ using document root <none> and password file <none>
I20260925 05:03:40.207578 11680 fs_manager.cc:359] Metadata directory not provided
I20260925 05:03:40.207708 11680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:03:40.208040 11680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:03:40 dist-test-slave-5n86 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312620, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for HTTP/127.30.247.20@KRBTEST.COM
I20260925 05:03:40.226663 11680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/data/instance:
uuid: "2660d74b41e74d39b19c376a1517e0a0"
format_stamp: "Formatted at 2026-09-25 05:03:40 on dist-test-slave-5n86"
server_key: "cfd984c95d98fdb0344ba5513665bf52"
server_key_iv: "3eeef34baf792794b4d68df0ec286d60"
server_key_version: "kuduclusterkey@0"
I20260925 05:03:40.229044 11680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance:
uuid: "2660d74b41e74d39b19c376a1517e0a0"
format_stamp: "Formatted at 2026-09-25 05:03:40 on dist-test-slave-5n86"
server_key: "cfd984c95d98fdb0344ba5513665bf52"
server_key_iv: "3eeef34baf792794b4d68df0ec286d60"
server_key_version: "kuduclusterkey@0"
I20260925 05:03:40.241688 11680 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.010s sys 0.001s
I20260925 05:03:40.264668 11700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:40.266589 11680 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.005s
I20260925 05:03:40.266830 11680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "2660d74b41e74d39b19c376a1517e0a0"
format_stamp: "Formatted at 2026-09-25 05:03:40 on dist-test-slave-5n86"
server_key: "cfd984c95d98fdb0344ba5513665bf52"
server_key_iv: "3eeef34baf792794b4d68df0ec286d60"
server_key_version: "kuduclusterkey@0"
I20260925 05:03:40.267135 11680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:03:40.312721 11680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:03:40.317595 11680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:03:40.317943 11680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:03:40.319103 11680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:03:40.320842 11680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:03:40.320968 11680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:40.321087 11680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:03:40.321147 11680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:40.393649 11680 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:45597
I20260925 05:03:40.393695 11812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:45597 every 8 connection(s)
I20260925 05:03:40.395370 11680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:03:40.397718 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 11680
I20260925 05:03:40.397918 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance
I20260925 05:03:40.398695 31708 external_mini_cluster.cc:1428] Setting key e5f3aee377b2d79a1e618f7b1c4f9578
Sep 25 05:03:40 dist-test-slave-5n86 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312620, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.247.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.2@KRBTEST.COM" created.
I20260925 05:03:40.455745 11816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:40.400155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:39663 (local address 127.30.247.62:46641)
0925 05:03:40.426776 (+ 26621us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:40.426782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:40.426802 (+ 20us) server_negotiation.cc:407] Connection header received
0925 05:03:40.426875 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:40.426880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:40.426970 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:40.427074 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:40.435744 (+ 8670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:40.436919 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:40.439349 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:40.439981 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:40.440073 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:40.447187 (+ 7114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:40.447194 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:40.447199 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:40.447242 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:40.450940 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:40.451868 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:40.451875 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:40.451878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:40.452034 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:40.452637 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:40.452643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:40.452646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:40.453066 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:40.453216 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:40.455138 (+ 1922us) server_negotiation.cc:299] Negotiation successful
0925 05:03:40.455364 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26479,"thread_start_us":32558,"threads_started":1}
I20260925 05:03:40.458355 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:40.399527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:46641 (local address 127.30.247.1:39663)
0925 05:03:40.423099 (+ 23572us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:40.423165 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:40.425972 (+ 2807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:40.433190 (+ 7218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:40.433203 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:40.433889 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:03:40.434778 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:40.434804 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:40.437133 (+ 2329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:40.437139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:40.439174 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:40.439190 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:40.440426 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:40.440433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:40.440740 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:40.441561 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:03:40.441590 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:03:40.444439 (+ 2849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:03:40.451109 (+ 6670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:40.451120 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:40.451140 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:40.451705 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:40.452184 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:40.452189 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:40.452193 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:40.452465 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:40.454230 (+ 1765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:03:40.454240 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:03:40.454689 (+ 449us) client_negotiation.cc:769] Sending connection context
0925 05:03:40.456422 (+ 1733us) client_negotiation.cc:240] Negotiation successful
0925 05:03:40.457004 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23167,"thread_start_us":27552,"threads_started":1}
I20260925 05:03:40.460443 11813 heartbeater.cc:347] Connected to a master server at 127.30.247.62:46641
I20260925 05:03:40.460791 11813 heartbeater.cc:464] Registering TS with master...
I20260925 05:03:40.461591 11813 heartbeater.cc:511] Master 127.30.247.62:46641 requested a full tablet report, sending...
I20260925 05:03:40.463956 11600 ts_manager.cc:194] Registered new tserver with Master: 2660d74b41e74d39b19c376a1517e0a0 (127.30.247.1:45597)
I20260925 05:03:40.465934 11600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.1@KRBTEST.COM'} at 127.30.247.1:39663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
I20260925 05:03:40.488929 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.247.20:46799/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:0
--local_ip_for_outbound_sockets=127.30.247.2
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:46641
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:03:40 dist-test-slave-5n86 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312620, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for HTTP/127.30.247.20@KRBTEST.COM
Sep 25 05:03:40 dist-test-slave-5n86 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312620, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for HTTP/127.30.247.20@KRBTEST.COM
W20260925 05:03:40.984900 11819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:03:40.985368 11819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:03:40.985551 11819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:03:40.998406 11819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:03:40.998607 11819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:03:40.998848 11819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:03:41.015009 11819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.30.247.20:46799/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:46641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:03:41.017812 11819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:03:41.019476 11819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:03:41.066860 11827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:41.067884 11826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:03:41.108884 11819 server_base.cc:1034] running on GCE node
W20260925 05:03:41.113020 11829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:03:41.114157 11819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:03:41.137405 11819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:03:41.145308 11819 hybrid_clock.cc:648] HybridClock initialized: now 1790312621145200 us; error 104 us; skew 500 ppm
I20260925 05:03:41.145838 11819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:03:41 dist-test-slave-5n86 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312621, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:03:41.151712 11819 init.cc:377] Logged in from keytab as kudu/127.30.247.2@KRBTEST.COM (short username kudu)
I20260925 05:03:41.171195 11819 webserver.cc:460] Webserver started at http://127.30.247.2:35403/ using document root <none> and password file <none>
I20260925 05:03:41.172289 11819 fs_manager.cc:359] Metadata directory not provided
I20260925 05:03:41.172508 11819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:03:41.172977 11819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:03:41 dist-test-slave-5n86 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312621, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for HTTP/127.30.247.20@KRBTEST.COM
I20260925 05:03:41.198451 11819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/data/instance:
uuid: "c4cf83bae1b44754b6903595609f4ff7"
format_stamp: "Formatted at 2026-09-25 05:03:41 on dist-test-slave-5n86"
server_key: "ee1c5cfe02b25373d1a1e45745c6afde"
server_key_iv: "43c7283803788f67d61c4ce6491f571e"
server_key_version: "kuduclusterkey@0"
I20260925 05:03:41.199617 11819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance:
uuid: "c4cf83bae1b44754b6903595609f4ff7"
format_stamp: "Formatted at 2026-09-25 05:03:41 on dist-test-slave-5n86"
server_key: "ee1c5cfe02b25373d1a1e45745c6afde"
server_key_iv: "43c7283803788f67d61c4ce6491f571e"
server_key_version: "kuduclusterkey@0"
I20260925 05:03:41.218734 11819 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.009s sys 0.007s
I20260925 05:03:41.251633 11838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:41.253546 11819 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.005s
I20260925 05:03:41.253819 11819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "c4cf83bae1b44754b6903595609f4ff7"
format_stamp: "Formatted at 2026-09-25 05:03:41 on dist-test-slave-5n86"
server_key: "ee1c5cfe02b25373d1a1e45745c6afde"
server_key_iv: "43c7283803788f67d61c4ce6491f571e"
server_key_version: "kuduclusterkey@0"
I20260925 05:03:41.254189 11819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:03:41.276353 11819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:03:41.280500 11819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:03:41.280874 11819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:03:41.281910 11819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:03:41.283360 11819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:03:41.283461 11819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:41.283620 11819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:03:41.283694 11819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:41.331799 11819 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:37937
I20260925 05:03:41.331802 11955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:37937 every 8 connection(s)
I20260925 05:03:41.333238 11819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:03:41.338299 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 11819
I20260925 05:03:41.338508 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance
I20260925 05:03:41.339243 31708 external_mini_cluster.cc:1428] Setting key c43676d428987959fb8bce7d6fec85f4
Sep 25 05:03:41 dist-test-slave-5n86 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312621, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:03:41.378000 11958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:41.336357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:58135 (local address 127.30.247.62:46641)
0925 05:03:41.354273 (+ 17916us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:41.354280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:41.354308 (+ 28us) server_negotiation.cc:407] Connection header received
0925 05:03:41.356783 (+ 2475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:41.356791 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:41.356879 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:41.359040 (+ 2161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:41.359205 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:41.360404 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:41.363010 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:41.363725 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:41.363873 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:41.367973 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:41.367982 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:41.367990 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:41.368034 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:41.371940 (+ 3906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:41.373002 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:41.373009 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:41.373016 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:41.373160 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:41.373690 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:41.373697 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:41.373703 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:41.374083 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:41.374236 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:41.377224 (+ 2988us) server_negotiation.cc:299] Negotiation successful
0925 05:03:41.377499 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17759,"thread_start_us":17951,"threads_started":1}
I20260925 05:03:41.378500 11959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:41.337331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:46641 (local address 127.30.247.2:58135)
0925 05:03:41.353417 (+ 16086us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:41.353466 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:41.355431 (+ 1965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:41.357301 (+ 1870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:41.357316 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:41.358001 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:03:41.358857 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:41.358885 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:41.360615 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:41.360622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:41.362837 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:41.362855 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:41.364183 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:41.364192 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:41.364523 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:41.365373 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:03:41.365407 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:03:41.367752 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:03:41.372238 (+ 4486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:41.372245 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:41.372266 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:41.372812 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:41.373297 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:41.373305 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:41.373311 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:41.373552 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:41.374319 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:03:41.374331 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:03:41.374796 (+ 465us) client_negotiation.cc:769] Sending connection context
0925 05:03:41.375100 (+ 304us) client_negotiation.cc:240] Negotiation successful
0925 05:03:41.375686 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15777,"thread_start_us":15528,"threads_started":1}
I20260925 05:03:41.380177 11956 heartbeater.cc:347] Connected to a master server at 127.30.247.62:46641
I20260925 05:03:41.380515 11956 heartbeater.cc:464] Registering TS with master...
I20260925 05:03:41.381433 11956 heartbeater.cc:511] Master 127.30.247.62:46641 requested a full tablet report, sending...
I20260925 05:03:41.383155 11600 ts_manager.cc:194] Registered new tserver with Master: c4cf83bae1b44754b6903595609f4ff7 (127.30.247.2:37937)
I20260925 05:03:41.383955 11600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.2@KRBTEST.COM'} at 127.30.247.2:58135
WARNING: no policy specified for kudu/127.30.247.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
I20260925 05:03:41.423540 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.247.20:46799/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:0
--local_ip_for_outbound_sockets=127.30.247.3
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:46641
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:03:41.469506 11813 heartbeater.cc:503] Master 127.30.247.62:46641 was elected leader, sending a full tablet report...
W20260925 05:03:41.790321 11962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:03:41.790676 11962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:03:41.790786 11962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:03:41.797709 11962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:03:41.797851 11962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:03:41.797999 11962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:03:41.807101 11962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.30.247.20:46799/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:46641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:03:41.808158 11962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:03:41.809351 11962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:03:41.821628 11967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:41.821671 11968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:41.821686 11970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:03:41.822775 11962 server_base.cc:1034] running on GCE node
I20260925 05:03:41.823287 11962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:03:41.824108 11962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:03:41.825311 11962 hybrid_clock.cc:648] HybridClock initialized: now 1790312621825261 us; error 62 us; skew 500 ppm
I20260925 05:03:41.825628 11962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:03:41 dist-test-slave-5n86 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312621, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:03:41.828677 11962 init.cc:377] Logged in from keytab as kudu/127.30.247.3@KRBTEST.COM (short username kudu)
I20260925 05:03:41.830076 11962 webserver.cc:460] Webserver started at http://127.30.247.3:42005/ using document root <none> and password file <none>
I20260925 05:03:41.830734 11962 fs_manager.cc:359] Metadata directory not provided
I20260925 05:03:41.830875 11962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:03:41.831171 11962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:03:41 dist-test-slave-5n86 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312621, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for HTTP/127.30.247.20@KRBTEST.COM
I20260925 05:03:41.845781 11962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/data/instance:
uuid: "060ca4e24c53452e833825f270745ac8"
format_stamp: "Formatted at 2026-09-25 05:03:41 on dist-test-slave-5n86"
server_key: "cf59a35d43efdcc034965b364c78384a"
server_key_iv: "d3d75e3718a92ab3f19dcc80ecd1b3de"
server_key_version: "kuduclusterkey@0"
I20260925 05:03:41.846735 11962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance:
uuid: "060ca4e24c53452e833825f270745ac8"
format_stamp: "Formatted at 2026-09-25 05:03:41 on dist-test-slave-5n86"
server_key: "cf59a35d43efdcc034965b364c78384a"
server_key_iv: "d3d75e3718a92ab3f19dcc80ecd1b3de"
server_key_version: "kuduclusterkey@0"
I20260925 05:03:41.851853 11962 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 05:03:41.867022 11977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:41.868687 11962 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260925 05:03:41.868870 11962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "060ca4e24c53452e833825f270745ac8"
format_stamp: "Formatted at 2026-09-25 05:03:41 on dist-test-slave-5n86"
server_key: "cf59a35d43efdcc034965b364c78384a"
server_key_iv: "d3d75e3718a92ab3f19dcc80ecd1b3de"
server_key_version: "kuduclusterkey@0"
I20260925 05:03:41.869112 11962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:03:41.885903 11962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:03:41.890110 11962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:03:41.890419 11962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:03:41.891520 11962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:03:41.892956 11962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:03:41.893059 11962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:41.893205 11962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:03:41.893278 11962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:41.926628 11962 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:37201
I20260925 05:03:41.926646 12089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:37201 every 8 connection(s)
I20260925 05:03:41.927884 11962 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:03:41.931851 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 11962
I20260925 05:03:41.932015 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance
I20260925 05:03:41.932504 31708 external_mini_cluster.cc:1428] Setting key e573897769c5f6ea1ebc711c66521260
Sep 25 05:03:41 dist-test-slave-5n86 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312621, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:03:41.948127 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:41.930632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:42663 (local address 127.30.247.62:46641)
0925 05:03:41.931160 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:41.931167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:41.932084 (+ 917us) server_negotiation.cc:407] Connection header received
0925 05:03:41.934535 (+ 2451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:41.934540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:41.934615 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:41.934730 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:41.936138 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:41.936932 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:41.938433 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:41.939061 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:41.939185 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:41.942404 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:41.942408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:41.942412 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:41.942442 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:41.944890 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:41.945611 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:41.945615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:41.945617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:41.945733 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:41.946248 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:41.946253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:41.946257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:41.946613 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:41.946739 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:41.947629 (+ 890us) server_negotiation.cc:299] Negotiation successful
0925 05:03:41.947885 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":243,"threads_started":1}
I20260925 05:03:41.949013 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:41.931174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:46641 (local address 127.30.247.3:42663)
0925 05:03:41.931930 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:41.931985 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:41.934256 (+ 2271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:41.934923 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:41.934931 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:41.935458 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:03:41.935990 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:41.936006 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:41.937054 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:41.937058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:41.938285 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:41.938296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:41.939306 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:41.939311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:41.939509 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:41.940200 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:03:41.940221 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:03:41.942205 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:03:41.945011 (+ 2806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:41.945018 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:41.945037 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:41.945460 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:41.945845 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:41.945851 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:41.945854 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:41.946107 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:41.946779 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:03:41.946787 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:03:41.947179 (+ 392us) client_negotiation.cc:769] Sending connection context
0925 05:03:41.947450 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 05:03:41.947922 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":307,"threads_started":1}
I20260925 05:03:41.950356 12090 heartbeater.cc:347] Connected to a master server at 127.30.247.62:46641
I20260925 05:03:41.950649 12090 heartbeater.cc:464] Registering TS with master...
I20260925 05:03:41.951318 12090 heartbeater.cc:511] Master 127.30.247.62:46641 requested a full tablet report, sending...
I20260925 05:03:41.952687 11600 ts_manager.cc:194] Registered new tserver with Master: 060ca4e24c53452e833825f270745ac8 (127.30.247.3:37201)
I20260925 05:03:41.953429 11600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.3@KRBTEST.COM'} at 127.30.247.3:42663
I20260925 05:03:41.960518 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:03:41.962739 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 11680
Sep 25 05:03:41 dist-test-slave-5n86 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.1@KRBTEST.COM
I20260925 05:03:41.980492 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:41.963681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51736 (local address 127.30.247.1:45597)
0925 05:03:41.964949 (+ 1268us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:41.964958 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:41.964985 (+ 27us) server_negotiation.cc:407] Connection header received
0925 05:03:41.965056 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:41.965065 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:41.965354 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:41.965534 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:41.966951 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:41.967943 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:41.969498 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:41.970047 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:41.970209 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:41.974872 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:41.974882 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:41.974895 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:41.974925 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:41.977712 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:41.978332 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:41.978346 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:41.978348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:41.978478 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:41.978801 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:41.978804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:41.978806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:41.979098 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:41.979520 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:41.979828 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 05:03:41.980098 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1095,"thread_start_us":916,"threads_started":1}
I20260925 05:03:41.981719 11808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:03:42.062793 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 11680
I20260925 05:03:42.078378 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 11819
Sep 25 05:03:42 dist-test-slave-5n86 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.2@KRBTEST.COM
I20260925 05:03:42.095198 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:42.079337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37682 (local address 127.30.247.2:37937)
0925 05:03:42.079922 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:42.079929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:42.079955 (+ 26us) server_negotiation.cc:407] Connection header received
0925 05:03:42.080013 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:42.080019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:42.080275 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:42.080408 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:42.081785 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:42.082831 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:42.084165 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:42.084886 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:42.084994 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:42.089353 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:42.089364 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:42.089378 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:42.089410 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:42.092417 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:42.093139 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:42.093153 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:42.093155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:42.093290 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:42.093662 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:42.093666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:42.093668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:42.093980 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:42.094295 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:42.094570 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 05:03:42.094837 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":312,"threads_started":1}
I20260925 05:03:42.096514 11951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:03:42.181373 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 11819
I20260925 05:03:42.197335 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 11962
Sep 25 05:03:42 dist-test-slave-5n86 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.3@KRBTEST.COM
I20260925 05:03:42.213099 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:42.198216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47174 (local address 127.30.247.3:37201)
0925 05:03:42.198965 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:42.198972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:42.199001 (+ 29us) server_negotiation.cc:407] Connection header received
0925 05:03:42.199059 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:42.199065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:42.199287 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:42.199422 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:42.200809 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:42.201798 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:42.203008 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:42.203549 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:42.203640 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:42.207697 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:42.207708 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:42.207721 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:42.207753 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:42.210336 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:42.211014 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:42.211022 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:42.211024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:42.211142 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:42.211514 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:42.211520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:42.211523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:42.211839 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:42.212151 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:42.212452 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 05:03:42.212743 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":441,"threads_started":1}
I20260925 05:03:42.214257 12085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:03:42.296483 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 11962
I20260925 05:03:42.312006 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 11568
I20260925 05:03:42.313382 11630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:03:42.419615 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 11568
I20260925 05:03:42.440853 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:46641
--webserver_interface=127.30.247.62
--webserver_port=33929
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:46641
--ranger_config_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.247.20:46799/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.247.20:46799/kms with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:03:42.769021 12102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:03:42.769320 12102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:03:42.769474 12102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:03:42.776394 12102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:03:42.776573 12102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:03:42.776687 12102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:03:42.776793 12102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:03:42.776893 12102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:03:42.786260 12102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.247.20:46799/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:46641
--ranger_config_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:46641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=33929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:03:42.787416 12102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:03:42.788666 12102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:03:42.800036 12108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:42.800031 12107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:42.800099 12110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:03:42.800951 12102 server_base.cc:1034] running on GCE node
I20260925 05:03:42.801700 12102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:03:42.802958 12102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:03:42.804186 12102 hybrid_clock.cc:648] HybridClock initialized: now 1790312622804134 us; error 60 us; skew 500 ppm
I20260925 05:03:42.804528 12102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:03:42 dist-test-slave-5n86 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312622, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:03:42.807554 12102 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:03:42.808931 12102 webserver.cc:460] Webserver started at http://127.30.247.62:33929/ using document root <none> and password file <none>
I20260925 05:03:42.809541 12102 fs_manager.cc:359] Metadata directory not provided
I20260925 05:03:42.809680 12102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 05:03:42 dist-test-slave-5n86 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312622, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for HTTP/127.30.247.20@KRBTEST.COM
I20260925 05:03:42.878337 12102 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 05:03:42.881500 12118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:42.883041 12102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:03:42.883237 12102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "a25c638006f44a2eafeb5c8625f882d4"
format_stamp: "Formatted at 2026-09-25 05:03:38 on dist-test-slave-5n86"
server_key: "b13f5a5e4da2f2fa4b0fabe291023cc4"
server_key_iv: "7e703214b0cc72653796f61307987753"
server_key_version: "kuduclusterkey@0"
I20260925 05:03:42.884222 12102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:03:42.927219 12102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:03:42.931092 12102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:03:42.931385 12102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:03:42.949894 12102 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:46641
I20260925 05:03:42.949923 12169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:46641 every 8 connection(s)
I20260925 05:03:42.951568 12102 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:03:42.952385 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 12102
I20260925 05:03:42.953482 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.247.20:46799/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:45597
--local_ip_for_outbound_sockets=127.30.247.1
--tserver_master_addrs=127.30.247.62:46641
--webserver_port=34201
--webserver_interface=127.30.247.1
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.247.20:46799/kms with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:03:42.958472 12170 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:03:42.961607 12170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4: Bootstrap starting.
I20260925 05:03:42.979568 12170 log.cc:824] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4: Log is configured to *not* fsync() on all Append() calls
I20260925 05:03:42.988118 12170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:03:42.988766 12170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4: Bootstrap complete.
I20260925 05:03:42.998142 12170 raft_consensus.cc:348] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a25c638006f44a2eafeb5c8625f882d4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46641 } }
I20260925 05:03:42.998994 12170 raft_consensus.cc:729] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a25c638006f44a2eafeb5c8625f882d4, State: Initialized, Role: FOLLOWER
I20260925 05:03:42.999575 12170 consensus_queue.cc:260] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a25c638006f44a2eafeb5c8625f882d4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46641 } }
I20260925 05:03:42.999819 12170 raft_consensus.cc:388] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:03:42.999950 12170 raft_consensus.cc:482] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:03:43.000120 12170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [term 1 FOLLOWER]: Advancing to term 2
I20260925 05:03:43.003890 12170 raft_consensus.cc:504] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a25c638006f44a2eafeb5c8625f882d4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46641 } }
I20260925 05:03:43.004374 12170 leader_election.cc:302] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a25c638006f44a2eafeb5c8625f882d4; no voters:
I20260925 05:03:43.005080 12170 leader_election.cc:288] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 05:03:43.005232 12174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 05:03:43.006315 12174 raft_consensus.cc:686] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [term 2 LEADER]: Becoming Leader. State: Replica: a25c638006f44a2eafeb5c8625f882d4, State: Running, Role: LEADER
I20260925 05:03:43.006932 12174 consensus_queue.cc:237] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a25c638006f44a2eafeb5c8625f882d4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46641 } }
I20260925 05:03:43.007246 12170 sys_catalog.cc:564] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:03:43.009320 12176 sys_catalog.cc:455] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a25c638006f44a2eafeb5c8625f882d4. Latest consensus state: current_term: 2 leader_uuid: "a25c638006f44a2eafeb5c8625f882d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a25c638006f44a2eafeb5c8625f882d4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46641 } } }
I20260925 05:03:43.009688 12176 sys_catalog.cc:458] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [sys.catalog]: This master's current role is: LEADER
I20260925 05:03:43.010195 12175 sys_catalog.cc:455] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a25c638006f44a2eafeb5c8625f882d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a25c638006f44a2eafeb5c8625f882d4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46641 } } }
I20260925 05:03:43.010490 12175 sys_catalog.cc:458] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4 [sys.catalog]: This master's current role is: LEADER
I20260925 05:03:43.016649 12180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:03:43.023383 12180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:03:43.025180 12180 catalog_manager.cc:1259] Loaded cluster ID: 7f46ba2ee4e6438bbfae9eb69da74b6b
I20260925 05:03:43.025355 12180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:03:43.030481 12180 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:03:43.032351 12180 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4: Loaded TSK: 0
I20260925 05:03:43.034199 12180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:03:43.038938 12170 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260925 05:03:43.388756 12172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:03:43.389206 12172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:03:43.389405 12172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:03:43.401521 12172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:03:43.401670 12172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:03:43.401950 12172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:03:43.417793 12172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.247.20:46799/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:45597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=34201
--tserver_master_addrs=127.30.247.62:46641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:03:43.419730 12172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:03:43.421461 12172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:03:43.475744 12208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:43.476521 12207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:43.479008 12210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:03:43.479781 12172 server_base.cc:1034] running on GCE node
I20260925 05:03:43.480789 12172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:03:43.482861 12172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:03:43.486267 12172 hybrid_clock.cc:648] HybridClock initialized: now 1790312623486100 us; error 149 us; skew 500 ppm
I20260925 05:03:43.486716 12172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:03:43 dist-test-slave-5n86 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312623, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:03:43.491091 12172 init.cc:377] Logged in from keytab as kudu/127.30.247.1@KRBTEST.COM (short username kudu)
I20260925 05:03:43.525926 12172 webserver.cc:460] Webserver started at http://127.30.247.1:34201/ using document root <none> and password file <none>
I20260925 05:03:43.527050 12172 fs_manager.cc:359] Metadata directory not provided
I20260925 05:03:43.527315 12172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 05:03:43 dist-test-slave-5n86 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312623, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for HTTP/127.30.247.20@KRBTEST.COM
I20260925 05:03:43.581404 12172 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.001s sys 0.004s
I20260925 05:03:43.594628 12219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:43.596928 12172 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.003s sys 0.004s
I20260925 05:03:43.597214 12172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "2660d74b41e74d39b19c376a1517e0a0"
format_stamp: "Formatted at 2026-09-25 05:03:40 on dist-test-slave-5n86"
server_key: "cfd984c95d98fdb0344ba5513665bf52"
server_key_iv: "3eeef34baf792794b4d68df0ec286d60"
server_key_version: "kuduclusterkey@0"
I20260925 05:03:43.598726 12172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:03:43.619773 12172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:03:43.624766 12172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:03:43.625111 12172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:03:43.626387 12172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:03:43.627733 12172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:03:43.627827 12172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:43.627983 12172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:03:43.628082 12172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:43.690976 12172 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:45597
I20260925 05:03:43.691023 12331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:45597 every 8 connection(s)
I20260925 05:03:43.692564 12172 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:03:43.701256 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 12172
I20260925 05:03:43.702065 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.247.20:46799/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:37937
--local_ip_for_outbound_sockets=127.30.247.2
--tserver_master_addrs=127.30.247.62:46641
--webserver_port=35403
--webserver_interface=127.30.247.2
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.247.20:46799/kms with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:03:43 dist-test-slave-5n86 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312623, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:03:43.746595 12334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:43.695888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:46641 (local address 127.30.247.1:45945)
0925 05:03:43.716838 (+ 20950us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:43.716891 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:43.719276 (+ 2385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:43.721705 (+ 2429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:43.721716 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:43.722337 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:03:43.723062 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:43.723086 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:43.725511 (+ 2425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:43.725518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:43.727482 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:43.727497 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:43.729228 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:43.729234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:43.729488 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:43.730283 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:03:43.730312 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:03:43.732925 (+ 2613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:03:43.741872 (+ 8947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:43.741884 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:43.741900 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:43.742548 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:43.743061 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:43.743065 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:43.743068 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:43.743307 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:43.744236 (+ 929us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:03:43.744244 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:03:43.744627 (+ 383us) client_negotiation.cc:769] Sending connection context
0925 05:03:43.744910 (+ 283us) client_negotiation.cc:240] Negotiation successful
0925 05:03:43.745414 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20645,"thread_start_us":20460,"threads_started":1}
I20260925 05:03:43.749590 12335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:43.695781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:45945 (local address 127.30.247.62:46641)
0925 05:03:43.717375 (+ 21594us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:43.717393 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:43.717451 (+ 58us) server_negotiation.cc:407] Connection header received
0925 05:03:43.720094 (+ 2643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:43.720113 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:43.720639 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:43.721028 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:43.723279 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:43.725317 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:43.727647 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:43.728692 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:43.728876 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:43.733189 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:43.733199 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:43.733218 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:43.733263 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:43.741660 (+ 8397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:43.742710 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:43.742719 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:43.742727 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:43.742919 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:43.743451 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:43.743456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:43.743459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:43.744080 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:43.747765 (+ 3685us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:43.748028 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 05:03:43.748402 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15018,"thread_start_us":20038,"threads_started":1}
I20260925 05:03:43.752998 12332 heartbeater.cc:347] Connected to a master server at 127.30.247.62:46641
I20260925 05:03:43.753300 12332 heartbeater.cc:464] Registering TS with master...
I20260925 05:03:43.754113 12332 heartbeater.cc:511] Master 127.30.247.62:46641 requested a full tablet report, sending...
W20260925 05:03:43.757306 12332 heartbeater.cc:644] Failed to heartbeat to 127.30.247.62:46641 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 05:03:43.757532 12332 heartbeater.cc:464] Registering TS with master...
I20260925 05:03:43.757812 12332 heartbeater.cc:511] Master 127.30.247.62:46641 requested a full tablet report, sending...
I20260925 05:03:43.759179 12332 heartbeater.cc:464] Registering TS with master...
I20260925 05:03:43.759441 12332 heartbeater.cc:511] Master 127.30.247.62:46641 requested a full tablet report, sending...
W20260925 05:03:43.760687 12332 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 05:03:44.093556 12336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:03:44.093873 12336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:03:44.094039 12336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:03:44.102856 12336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:03:44.103005 12336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:03:44.103171 12336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:03:44.113446 12336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.247.20:46799/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:37937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=35403
--tserver_master_addrs=127.30.247.62:46641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:03:44.114732 12336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:03:44.116014 12336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:03:44.132511 12350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:44.135331 12353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:44.135862 12351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:03:44.138294 12336 server_base.cc:1034] running on GCE node
I20260925 05:03:44.138972 12336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:03:44.140039 12336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:03:44.141296 12336 hybrid_clock.cc:648] HybridClock initialized: now 1790312624141219 us; error 62 us; skew 500 ppm
I20260925 05:03:44.141662 12336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:03:44 dist-test-slave-5n86 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312624, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:03:44.155179 12336 init.cc:377] Logged in from keytab as kudu/127.30.247.2@KRBTEST.COM (short username kudu)
I20260925 05:03:44.176323 12336 webserver.cc:460] Webserver started at http://127.30.247.2:35403/ using document root <none> and password file <none>
I20260925 05:03:44.177556 12336 fs_manager.cc:359] Metadata directory not provided
I20260925 05:03:44.177742 12336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 05:03:44 dist-test-slave-5n86 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312624, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for HTTP/127.30.247.20@KRBTEST.COM
I20260925 05:03:44.222431 12336 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.001s sys 0.008s
I20260925 05:03:44.226397 12363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:44.228299 12336 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 05:03:44.228477 12336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "c4cf83bae1b44754b6903595609f4ff7"
format_stamp: "Formatted at 2026-09-25 05:03:41 on dist-test-slave-5n86"
server_key: "ee1c5cfe02b25373d1a1e45745c6afde"
server_key_iv: "43c7283803788f67d61c4ce6491f571e"
server_key_version: "kuduclusterkey@0"
I20260925 05:03:44.230051 12336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 25 05:03:44 dist-test-slave-5n86 krb5kdc[11124](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790312624, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:03:44.281277 12336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:03:44.286469 12336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:03:44.286823 12336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:03:44.288105 12336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:03:44.289882 12336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:03:44.289994 12336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:44.290107 12336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:03:44.290184 12336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:44.352219 12336 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:37937
I20260925 05:03:44.352268 12476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:37937 every 8 connection(s)
I20260925 05:03:44.353827 12336 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:03:44.357754 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 12336
I20260925 05:03:44.358518 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.247.20:46799/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:37201
--local_ip_for_outbound_sockets=127.30.247.3
--tserver_master_addrs=127.30.247.62:46641
--webserver_port=42005
--webserver_interface=127.30.247.3
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.247.20:46799/kms with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:03:44 dist-test-slave-5n86 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312624, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:03:44.394771 12480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:44.359656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:37783 (local address 127.30.247.62:46641)
0925 05:03:44.379818 (+ 20162us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:44.379828 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:44.379860 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:03:44.379964 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:44.379973 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:44.380091 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:44.380247 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:44.382250 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:44.382977 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:44.384909 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:44.385334 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:44.385472 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:44.389032 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:44.389037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:44.389040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:44.389072 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:44.391790 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:44.392481 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:44.392486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:44.392490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:44.392672 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:44.393103 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:44.393107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:44.393111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:44.393370 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:44.393519 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:44.394198 (+ 679us) server_negotiation.cc:299] Negotiation successful
0925 05:03:44.394402 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19955,"thread_start_us":20701,"threads_started":1}
I20260925 05:03:44.395946 12479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:44.358908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:46641 (local address 127.30.247.2:37783)
0925 05:03:44.360409 (+ 1501us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:44.360464 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:44.363574 (+ 3110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:44.380642 (+ 17068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:44.380662 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:44.381409 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:03:44.382057 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:44.382076 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:44.383140 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:44.383148 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:44.384744 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:44.384763 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:44.385586 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:44.385593 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:44.385839 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:44.386529 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:03:44.386552 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:03:44.388808 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:03:44.391934 (+ 3126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:44.391942 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:44.391958 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:44.392349 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:44.392776 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:44.392782 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:44.392786 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:44.392987 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:44.393514 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:03:44.393521 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:03:44.393829 (+ 308us) client_negotiation.cc:769] Sending connection context
0925 05:03:44.394085 (+ 256us) client_negotiation.cc:240] Negotiation successful
0925 05:03:44.394611 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1194,"thread_start_us":945,"threads_started":1}
I20260925 05:03:44.397485 12477 heartbeater.cc:347] Connected to a master server at 127.30.247.62:46641
I20260925 05:03:44.397791 12477 heartbeater.cc:464] Registering TS with master...
I20260925 05:03:44.398521 12477 heartbeater.cc:511] Master 127.30.247.62:46641 requested a full tablet report, sending...
I20260925 05:03:44.400275 12135 ts_manager.cc:194] Registered new tserver with Master: c4cf83bae1b44754b6903595609f4ff7 (127.30.247.2:37937)
I20260925 05:03:44.401991 12135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.2@KRBTEST.COM'} at 127.30.247.2:37783
Sep 25 05:03:44 dist-test-slave-5n86 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312624, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for HTTP/127.30.247.20@KRBTEST.COM
I20260925 05:03:44.760812 12332 heartbeater.cc:464] Registering TS with master...
I20260925 05:03:44.761240 12332 heartbeater.cc:511] Master 127.30.247.62:46641 requested a full tablet report, sending...
I20260925 05:03:44.762610 12135 ts_manager.cc:194] Registered new tserver with Master: 2660d74b41e74d39b19c376a1517e0a0 (127.30.247.1:45597)
I20260925 05:03:44.763813 12135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.1@KRBTEST.COM'} at 127.30.247.1:45945
W20260925 05:03:44.798487 12481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:03:44.798940 12481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:03:44.799124 12481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:03:44.810173 12481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:03:44.810395 12481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:03:44.810619 12481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:03:44.825440 12481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.247.20:46799/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:37201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=42005
--tserver_master_addrs=127.30.247.62:46641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:03:44.827241 12481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:03:44.828904 12481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:03:44.873122 12487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:44.881384 12486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:03:44.882156 12481 server_base.cc:1034] running on GCE node
W20260925 05:03:44.882629 12489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:03:44.883466 12481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:03:44.889983 12481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:03:44.891227 12481 hybrid_clock.cc:648] HybridClock initialized: now 1790312624891129 us; error 95 us; skew 500 ppm
I20260925 05:03:44.891646 12481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:03:44 dist-test-slave-5n86 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312624, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:03:44.895682 12481 init.cc:377] Logged in from keytab as kudu/127.30.247.3@KRBTEST.COM (short username kudu)
I20260925 05:03:44.915995 12481 webserver.cc:460] Webserver started at http://127.30.247.3:42005/ using document root <none> and password file <none>
I20260925 05:03:44.917037 12481 fs_manager.cc:359] Metadata directory not provided
I20260925 05:03:44.917258 12481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 05:03:44 dist-test-slave-5n86 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312624, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for HTTP/127.30.247.20@KRBTEST.COM
I20260925 05:03:44.962538 12481 fs_manager.cc:709] Time spent opening directory manager: real 0.019s user 0.005s sys 0.002s
I20260925 05:03:44.977761 12497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:44.985389 12481 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.004s sys 0.011s
I20260925 05:03:44.985669 12481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "060ca4e24c53452e833825f270745ac8"
format_stamp: "Formatted at 2026-09-25 05:03:41 on dist-test-slave-5n86"
server_key: "cf59a35d43efdcc034965b364c78384a"
server_key_iv: "d3d75e3718a92ab3f19dcc80ecd1b3de"
server_key_version: "kuduclusterkey@0"
I20260925 05:03:44.987241 12481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:03:45.040324 12481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:03:45.045457 12481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:03:45.045802 12481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:03:45.053081 12481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:03:45.060997 12481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:03:45.061173 12481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:45.061327 12481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:03:45.061447 12481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 05:03:45 dist-test-slave-5n86 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312624, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for HTTP/127.30.247.20@KRBTEST.COM
I20260925 05:03:45.406056 12477 heartbeater.cc:503] Master 127.30.247.62:46641 was elected leader, sending a full tablet report...
I20260925 05:03:45.439277 12481 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:37201
I20260925 05:03:45.439339 12617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:37201 every 8 connection(s)
I20260925 05:03:45.441098 12481 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:03:45.443915 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 12481
Sep 25 05:03:45 dist-test-slave-5n86 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312624, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:03:45.463120 12622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:45.444941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50654 (local address 127.30.247.62:46641)
0925 05:03:45.449827 (+ 4886us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:45.449833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:45.449851 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:03:45.449911 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:45.449917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:45.449991 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:45.450101 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:45.451749 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.452649 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.453962 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.454381 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.454523 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:45.457337 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:45.457343 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:45.457346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:45.457389 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:45.460468 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:45.461148 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:45.461154 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:45.461158 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:45.461305 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:45.461684 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:45.461688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:45.461693 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:45.461932 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:45.462212 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:45.462510 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 05:03:45.462716 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4756,"spinlock_wait_cycles":4864,"thread_start_us":4643,"threads_started":1}
I20260925 05:03:45.463336 12620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:45.443901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:53285 (local address 127.30.247.62:46641)
0925 05:03:45.444461 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:45.444468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:45.448628 (+ 4160us) server_negotiation.cc:407] Connection header received
0925 05:03:45.449143 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:45.449147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:45.449210 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:45.449331 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:45.451003 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.451717 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.453280 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.453695 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.453861 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:45.457197 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:45.457202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:45.457205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:45.457236 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:45.460373 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:45.461080 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:45.461085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:45.461088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:45.461199 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:45.461630 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:45.461635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:45.461638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:45.461916 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:45.462061 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:45.462686 (+ 625us) server_negotiation.cc:299] Negotiation successful
0925 05:03:45.462886 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":427,"thread_start_us":264,"threads_started":1}
I20260925 05:03:45.464391 12621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:45.444473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:46641 (local address 127.30.247.3:53285)
0925 05:03:45.445419 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:45.445478 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:45.448696 (+ 3218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:45.449621 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:45.449634 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:45.450280 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:03:45.450844 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:45.450861 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.451871 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.451877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:45.453146 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:45.453157 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.453940 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.453947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:45.454176 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:45.454831 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:03:45.454851 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:03:45.456988 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:03:45.460521 (+ 3533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:45.460529 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:45.460568 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:45.460957 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:45.461312 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:45.461318 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:45.461322 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:45.461512 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:45.462036 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:03:45.462043 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:03:45.462346 (+ 303us) client_negotiation.cc:769] Sending connection context
0925 05:03:45.462589 (+ 243us) client_negotiation.cc:240] Negotiation successful
0925 05:03:45.463073 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":342,"threads_started":1}
I20260925 05:03:45.465934 12618 heartbeater.cc:347] Connected to a master server at 127.30.247.62:46641
I20260925 05:03:45.466213 12618 heartbeater.cc:464] Registering TS with master...
I20260925 05:03:45.466857 12618 heartbeater.cc:511] Master 127.30.247.62:46641 requested a full tablet report, sending...
I20260925 05:03:45.468178 12134 ts_manager.cc:194] Registered new tserver with Master: 060ca4e24c53452e833825f270745ac8 (127.30.247.3:37201)
I20260925 05:03:45.469105 12134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.3@KRBTEST.COM'} at 127.30.247.3:53285
I20260925 05:03:45.477741 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:03:45.492842 12622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:45.480475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50664 (local address 127.30.247.62:46641)
0925 05:03:45.480643 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:45.480647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:45.481074 (+ 427us) server_negotiation.cc:407] Connection header received
0925 05:03:45.481197 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:45.481201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:45.481265 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:45.481342 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:45.482745 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.483384 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.484671 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.485062 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.485152 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:45.487722 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:45.487726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:45.487730 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:45.487760 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:45.490425 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:45.491073 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:45.491078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:45.491080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:45.491213 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:45.491612 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:45.491617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:45.491619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:45.491867 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:45.492052 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:45.492374 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 05:03:45.492577 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 05:03:45.497532 12134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:03:45.504880 12134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:03:45.529939 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:45.518514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51746 (local address 127.30.247.1:45597)
0925 05:03:45.521347 (+ 2833us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:45.521358 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:45.521389 (+ 31us) server_negotiation.cc:407] Connection header received
0925 05:03:45.521499 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:45.521507 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:45.521972 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:45.522934 (+ 962us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:03:45.523213 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.524879 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.527675 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.528716 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.528864 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:45.529103 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:45.529402 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 05:03:45.529571 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2654,"thread_start_us":2501,"threads_started":1}
I20260925 05:03:45.530062 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:45.518383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:45597 (local address 127.0.0.1:51746)
0925 05:03:45.520395 (+ 2012us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:45.520462 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:45.520679 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:45.522204 (+ 1525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:45.522211 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:45.522240 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:03:45.522711 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:45.522725 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.525051 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.525058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:45.527507 (+ 2449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:45.527518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.528964 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.528970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:45.529170 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:45.529213 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 05:03:45.529364 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 05:03:45.529542 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1704,"thread_start_us":1545,"threads_started":1}
I20260925 05:03:45.531528 12638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:45.521504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37688 (local address 127.30.247.2:37937)
0925 05:03:45.523885 (+ 2381us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:45.523893 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:45.523919 (+ 26us) server_negotiation.cc:407] Connection header received
0925 05:03:45.524025 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:45.524032 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:45.524243 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:45.524426 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:03:45.525087 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.526559 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.529073 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.530464 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.530621 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:45.530840 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:45.531037 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 05:03:45.531189 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2219,"thread_start_us":2052,"threads_started":1}
I20260925 05:03:45.532784 12637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:45.520924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:37937 (local address 127.0.0.1:37688)
0925 05:03:45.523246 (+ 2322us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:45.523265 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:45.523371 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:45.524434 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:45.524439 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:45.524463 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:03:45.524883 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:45.524898 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.526733 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.526737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:45.528888 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:45.528904 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.530690 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.530696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:45.530868 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:45.530900 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 05:03:45.530989 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 05:03:45.531111 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2209,"mutex_wait_us":2,"thread_start_us":2070,"threads_started":1}
I20260925 05:03:45.532992 12635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:45.518862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47184 (local address 127.30.247.3:37201)
0925 05:03:45.524196 (+ 5334us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:45.524202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:45.524223 (+ 21us) server_negotiation.cc:407] Connection header received
0925 05:03:45.524293 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:45.524298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:45.524519 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:45.524734 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:03:45.525538 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.527046 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.530655 (+ 3609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.532057 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.532196 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:45.532418 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:45.532488 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 05:03:45.532661 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5189,"thread_start_us":5063,"threads_started":1}
I20260925 05:03:45.532974 12633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:45.518447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:37201 (local address 127.0.0.1:47184)
0925 05:03:45.521554 (+ 3107us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:45.521570 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:45.521669 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:45.524734 (+ 3065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:45.524739 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:45.524763 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:03:45.525277 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:45.525294 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.527186 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.527191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:45.530476 (+ 3285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:45.530492 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.532238 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.532243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:45.532367 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:45.532388 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 05:03:45.532448 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 05:03:45.532522 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3009,"mutex_wait_us":47,"thread_start_us":3244,"threads_started":1}
I20260925 05:03:45.534338 12267 tablet_service.cc:1467] Processing CreateTablet for tablet ed184fd0cc0f46beba0c750cd1806aef (DEFAULT_TABLE table=test-table [id=e211352458444f5e986bd27d83201880]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:03:45.535719 12267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed184fd0cc0f46beba0c750cd1806aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:03:45.536278 12546 tablet_service.cc:1467] Processing CreateTablet for tablet ed184fd0cc0f46beba0c750cd1806aef (DEFAULT_TABLE table=test-table [id=e211352458444f5e986bd27d83201880]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:03:45.536298 12411 tablet_service.cc:1467] Processing CreateTablet for tablet ed184fd0cc0f46beba0c750cd1806aef (DEFAULT_TABLE table=test-table [id=e211352458444f5e986bd27d83201880]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:03:45.537573 12546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed184fd0cc0f46beba0c750cd1806aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:03:45.537587 12411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed184fd0cc0f46beba0c750cd1806aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:03:45.547535 12639 tablet_bootstrap.cc:492] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0: Bootstrap starting.
I20260925 05:03:45.550887 12639 tablet_bootstrap.cc:654] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0: Neither blocks nor log segments found. Creating new log.
I20260925 05:03:45.551970 12640 tablet_bootstrap.cc:492] T ed184fd0cc0f46beba0c750cd1806aef P c4cf83bae1b44754b6903595609f4ff7: Bootstrap starting.
I20260925 05:03:45.552126 12639 log.cc:824] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0: Log is configured to *not* fsync() on all Append() calls
I20260925 05:03:45.552752 12641 tablet_bootstrap.cc:492] T ed184fd0cc0f46beba0c750cd1806aef P 060ca4e24c53452e833825f270745ac8: Bootstrap starting.
I20260925 05:03:45.555251 12640 tablet_bootstrap.cc:654] T ed184fd0cc0f46beba0c750cd1806aef P c4cf83bae1b44754b6903595609f4ff7: Neither blocks nor log segments found. Creating new log.
I20260925 05:03:45.556175 12640 log.cc:824] T ed184fd0cc0f46beba0c750cd1806aef P c4cf83bae1b44754b6903595609f4ff7: Log is configured to *not* fsync() on all Append() calls
I20260925 05:03:45.556805 12641 tablet_bootstrap.cc:654] T ed184fd0cc0f46beba0c750cd1806aef P 060ca4e24c53452e833825f270745ac8: Neither blocks nor log segments found. Creating new log.
I20260925 05:03:45.557968 12641 log.cc:824] T ed184fd0cc0f46beba0c750cd1806aef P 060ca4e24c53452e833825f270745ac8: Log is configured to *not* fsync() on all Append() calls
I20260925 05:03:45.559125 12640 tablet_bootstrap.cc:492] T ed184fd0cc0f46beba0c750cd1806aef P c4cf83bae1b44754b6903595609f4ff7: No bootstrap required, opened a new log
I20260925 05:03:45.559128 12639 tablet_bootstrap.cc:492] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0: No bootstrap required, opened a new log
I20260925 05:03:45.559391 12639 ts_tablet_manager.cc:1397] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260925 05:03:45.559444 12640 ts_tablet_manager.cc:1397] T ed184fd0cc0f46beba0c750cd1806aef P c4cf83bae1b44754b6903595609f4ff7: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 05:03:45.561407 12641 tablet_bootstrap.cc:492] T ed184fd0cc0f46beba0c750cd1806aef P 060ca4e24c53452e833825f270745ac8: No bootstrap required, opened a new log
I20260925 05:03:45.561668 12641 ts_tablet_manager.cc:1397] T ed184fd0cc0f46beba0c750cd1806aef P 060ca4e24c53452e833825f270745ac8: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260925 05:03:45.568188 12641 raft_consensus.cc:348] T ed184fd0cc0f46beba0c750cd1806aef P 060ca4e24c53452e833825f270745ac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2660d74b41e74d39b19c376a1517e0a0" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45597 } } peers { permanent_uuid: "060ca4e24c53452e833825f270745ac8" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 37201 } } peers { permanent_uuid: "c4cf83bae1b44754b6903595609f4ff7" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 37937 } }
I20260925 05:03:45.568545 12641 raft_consensus.cc:374] T ed184fd0cc0f46beba0c750cd1806aef P 060ca4e24c53452e833825f270745ac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:03:45.568691 12641 raft_consensus.cc:729] T ed184fd0cc0f46beba0c750cd1806aef P 060ca4e24c53452e833825f270745ac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060ca4e24c53452e833825f270745ac8, State: Initialized, Role: FOLLOWER
I20260925 05:03:45.569449 12641 consensus_queue.cc:260] T ed184fd0cc0f46beba0c750cd1806aef P 060ca4e24c53452e833825f270745ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2660d74b41e74d39b19c376a1517e0a0" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45597 } } peers { permanent_uuid: "060ca4e24c53452e833825f270745ac8" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 37201 } } peers { permanent_uuid: "c4cf83bae1b44754b6903595609f4ff7" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 37937 } }
I20260925 05:03:45.569875 12640 raft_consensus.cc:348] T ed184fd0cc0f46beba0c750cd1806aef P c4cf83bae1b44754b6903595609f4ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2660d74b41e74d39b19c376a1517e0a0" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45597 } } peers { permanent_uuid: "060ca4e24c53452e833825f270745ac8" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 37201 } } peers { permanent_uuid: "c4cf83bae1b44754b6903595609f4ff7" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 37937 } }
I20260925 05:03:45.569888 12639 raft_consensus.cc:348] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2660d74b41e74d39b19c376a1517e0a0" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45597 } } peers { permanent_uuid: "060ca4e24c53452e833825f270745ac8" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 37201 } } peers { permanent_uuid: "c4cf83bae1b44754b6903595609f4ff7" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 37937 } }
I20260925 05:03:45.570282 12639 raft_consensus.cc:374] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:03:45.570365 12640 raft_consensus.cc:374] T ed184fd0cc0f46beba0c750cd1806aef P c4cf83bae1b44754b6903595609f4ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:03:45.570425 12639 raft_consensus.cc:729] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2660d74b41e74d39b19c376a1517e0a0, State: Initialized, Role: FOLLOWER
I20260925 05:03:45.570510 12640 raft_consensus.cc:729] T ed184fd0cc0f46beba0c750cd1806aef P c4cf83bae1b44754b6903595609f4ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4cf83bae1b44754b6903595609f4ff7, State: Initialized, Role: FOLLOWER
I20260925 05:03:45.570750 12618 heartbeater.cc:503] Master 127.30.247.62:46641 was elected leader, sending a full tablet report...
I20260925 05:03:45.571058 12639 consensus_queue.cc:260] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2660d74b41e74d39b19c376a1517e0a0" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45597 } } peers { permanent_uuid: "060ca4e24c53452e833825f270745ac8" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 37201 } } peers { permanent_uuid: "c4cf83bae1b44754b6903595609f4ff7" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 37937 } }
I20260925 05:03:45.571091 12640 consensus_queue.cc:260] T ed184fd0cc0f46beba0c750cd1806aef P c4cf83bae1b44754b6903595609f4ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2660d74b41e74d39b19c376a1517e0a0" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45597 } } peers { permanent_uuid: "060ca4e24c53452e833825f270745ac8" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 37201 } } peers { permanent_uuid: "c4cf83bae1b44754b6903595609f4ff7" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 37937 } }
I20260925 05:03:45.571527 12641 ts_tablet_manager.cc:1428] T ed184fd0cc0f46beba0c750cd1806aef P 060ca4e24c53452e833825f270745ac8: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260925 05:03:45.572789 12332 heartbeater.cc:503] Master 127.30.247.62:46641 was elected leader, sending a full tablet report...
I20260925 05:03:45.573004 12639 ts_tablet_manager.cc:1428] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260925 05:03:45.573217 12640 ts_tablet_manager.cc:1428] T ed184fd0cc0f46beba0c750cd1806aef P c4cf83bae1b44754b6903595609f4ff7: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260925 05:03:45.632138 12478 tablet.cc:2369] T ed184fd0cc0f46beba0c750cd1806aef P c4cf83bae1b44754b6903595609f4ff7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:03:45.690665 12647 raft_consensus.cc:482] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:03:45.690987 12647 raft_consensus.cc:504] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2660d74b41e74d39b19c376a1517e0a0" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45597 } } peers { permanent_uuid: "060ca4e24c53452e833825f270745ac8" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 37201 } } peers { permanent_uuid: "c4cf83bae1b44754b6903595609f4ff7" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 37937 } }
W20260925 05:03:45.692812 12619 tablet.cc:2369] T ed184fd0cc0f46beba0c750cd1806aef P 060ca4e24c53452e833825f270745ac8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:03:45.695017 12647 leader_election.cc:288] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 060ca4e24c53452e833825f270745ac8 (127.30.247.3:37201), c4cf83bae1b44754b6903595609f4ff7 (127.30.247.2:37937)
W20260925 05:03:45.695472 12333 tablet.cc:2369] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:03:45.701366 12635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:45.693611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:35119 (local address 127.30.247.3:37201)
0925 05:03:45.693835 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:45.693844 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:45.694321 (+ 477us) server_negotiation.cc:407] Connection header received
0925 05:03:45.694652 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:45.694661 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:45.694762 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:45.694848 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:03:45.696352 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.697517 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.699451 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.700383 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.700562 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:45.700632 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:45.700983 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 05:03:45.701115 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 05:03:45.701463 12648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:45.693508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:37201 (local address 127.30.247.1:35119)
0925 05:03:45.694221 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:45.694243 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:45.694440 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:45.695063 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:45.695072 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:45.695099 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:03:45.696174 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:45.696193 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.697713 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.697717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:45.699301 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:45.699313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.700620 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.700627 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:45.700810 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:45.700845 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 05:03:45.700938 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 05:03:45.701063 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":3,"thread_start_us":359,"threads_started":1}
I20260925 05:03:45.702409 12638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:45.694486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:42935 (local address 127.30.247.2:37937)
0925 05:03:45.694659 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:45.694665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:45.695190 (+ 525us) server_negotiation.cc:407] Connection header received
0925 05:03:45.695469 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:45.695474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:45.695559 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:45.695659 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:03:45.696459 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.697517 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.700027 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.701184 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.701343 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:45.701424 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:45.702056 (+ 632us) server_negotiation.cc:299] Negotiation successful
0925 05:03:45.702174 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 05:03:45.702543 12649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:45.694355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:37937 (local address 127.30.247.1:42935)
0925 05:03:45.695084 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:45.695101 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:45.695305 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:45.695712 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:45.695717 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:45.695742 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:03:45.696208 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:45.696218 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.697652 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.697657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:45.699874 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:45.699888 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.701655 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.701660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:45.701837 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:45.701876 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 05:03:45.702030 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 05:03:45.702154 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":3,"thread_start_us":443,"threads_started":1}
I20260925 05:03:45.702701 12568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed184fd0cc0f46beba0c750cd1806aef" candidate_uuid: "2660d74b41e74d39b19c376a1517e0a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060ca4e24c53452e833825f270745ac8" is_pre_election: true
I20260925 05:03:45.703248 12568 raft_consensus.cc:2454] T ed184fd0cc0f46beba0c750cd1806aef P 060ca4e24c53452e833825f270745ac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2660d74b41e74d39b19c376a1517e0a0 in term 0.
I20260925 05:03:45.703433 12431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed184fd0cc0f46beba0c750cd1806aef" candidate_uuid: "2660d74b41e74d39b19c376a1517e0a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4cf83bae1b44754b6903595609f4ff7" is_pre_election: true
I20260925 05:03:45.703907 12222 leader_election.cc:302] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060ca4e24c53452e833825f270745ac8, 2660d74b41e74d39b19c376a1517e0a0; no voters:
I20260925 05:03:45.704001 12431 raft_consensus.cc:2454] T ed184fd0cc0f46beba0c750cd1806aef P c4cf83bae1b44754b6903595609f4ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2660d74b41e74d39b19c376a1517e0a0 in term 0.
I20260925 05:03:45.704319 12647 raft_consensus.cc:2781] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:03:45.704469 12647 raft_consensus.cc:482] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:03:45.704607 12647 raft_consensus.cc:3037] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:03:45.706866 12647 raft_consensus.cc:504] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2660d74b41e74d39b19c376a1517e0a0" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45597 } } peers { permanent_uuid: "060ca4e24c53452e833825f270745ac8" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 37201 } } peers { permanent_uuid: "c4cf83bae1b44754b6903595609f4ff7" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 37937 } }
I20260925 05:03:45.707537 12647 leader_election.cc:288] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0 [CANDIDATE]: Term 1 election: Requested vote from peers 060ca4e24c53452e833825f270745ac8 (127.30.247.3:37201), c4cf83bae1b44754b6903595609f4ff7 (127.30.247.2:37937)
I20260925 05:03:45.707942 12568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed184fd0cc0f46beba0c750cd1806aef" candidate_uuid: "2660d74b41e74d39b19c376a1517e0a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060ca4e24c53452e833825f270745ac8"
I20260925 05:03:45.708019 12431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed184fd0cc0f46beba0c750cd1806aef" candidate_uuid: "2660d74b41e74d39b19c376a1517e0a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4cf83bae1b44754b6903595609f4ff7"
I20260925 05:03:45.708155 12568 raft_consensus.cc:3037] T ed184fd0cc0f46beba0c750cd1806aef P 060ca4e24c53452e833825f270745ac8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:03:45.708184 12431 raft_consensus.cc:3037] T ed184fd0cc0f46beba0c750cd1806aef P c4cf83bae1b44754b6903595609f4ff7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:03:45.711380 12568 raft_consensus.cc:2454] T ed184fd0cc0f46beba0c750cd1806aef P 060ca4e24c53452e833825f270745ac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2660d74b41e74d39b19c376a1517e0a0 in term 1.
I20260925 05:03:45.711380 12431 raft_consensus.cc:2454] T ed184fd0cc0f46beba0c750cd1806aef P c4cf83bae1b44754b6903595609f4ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2660d74b41e74d39b19c376a1517e0a0 in term 1.
I20260925 05:03:45.711728 12222 leader_election.cc:302] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060ca4e24c53452e833825f270745ac8, 2660d74b41e74d39b19c376a1517e0a0; no voters:
I20260925 05:03:45.712026 12647 raft_consensus.cc:2781] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:03:45.712402 12647 raft_consensus.cc:686] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0 [term 1 LEADER]: Becoming Leader. State: Replica: 2660d74b41e74d39b19c376a1517e0a0, State: Running, Role: LEADER
I20260925 05:03:45.712867 12647 consensus_queue.cc:237] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2660d74b41e74d39b19c376a1517e0a0" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45597 } } peers { permanent_uuid: "060ca4e24c53452e833825f270745ac8" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 37201 } } peers { permanent_uuid: "c4cf83bae1b44754b6903595609f4ff7" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 37937 } }
I20260925 05:03:45.717483 12133 catalog_manager.cc:5495] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2660d74b41e74d39b19c376a1517e0a0 (127.30.247.1). New cstate: current_term: 1 leader_uuid: "2660d74b41e74d39b19c376a1517e0a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2660d74b41e74d39b19c376a1517e0a0" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "060ca4e24c53452e833825f270745ac8" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 37201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4cf83bae1b44754b6903595609f4ff7" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 37937 } health_report { overall_health: UNKNOWN } } }
I20260925 05:03:45.766055 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:45.761005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51760 (local address 127.30.247.1:45597)
0925 05:03:45.761194 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:45.761198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:45.761215 (+ 17us) server_negotiation.cc:407] Connection header received
0925 05:03:45.761348 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:45.761351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:45.761434 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:45.761551 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:03:45.761892 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.762611 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.763994 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.764416 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.764521 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:45.765033 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:03:45.765562 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:03:45.765670 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:45.765796 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 05:03:45.765859 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 05:03:45.774288 12431 raft_consensus.cc:1261] T ed184fd0cc0f46beba0c750cd1806aef P c4cf83bae1b44754b6903595609f4ff7 [term 1 FOLLOWER]: Refusing update from remote peer 2660d74b41e74d39b19c376a1517e0a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:03:45.774288 12568 raft_consensus.cc:1261] T ed184fd0cc0f46beba0c750cd1806aef P 060ca4e24c53452e833825f270745ac8 [term 1 FOLLOWER]: Refusing update from remote peer 2660d74b41e74d39b19c376a1517e0a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:03:45.775401 12650 consensus_queue.cc:1035] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4cf83bae1b44754b6903595609f4ff7" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 37937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:03:45.775753 12647 consensus_queue.cc:1035] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "060ca4e24c53452e833825f270745ac8" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 37201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:03:45.789093 12654 mvcc.cc:205] Tried to move back new op lower bound from 7333120515160469504 to 7333120514923474944. Current Snapshot: MvccSnapshot[applied={T|T < 7333120515160469504}]
I20260925 05:03:45.801616 12133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:03:45.801929 12133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:03:45.805570 12133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a25c638006f44a2eafeb5c8625f882d4: Sending DeleteTablet for 3 replicas of tablet ed184fd0cc0f46beba0c750cd1806aef
I20260925 05:03:45.806617 12267 tablet_service.cc:1514] Processing DeleteTablet for tablet ed184fd0cc0f46beba0c750cd1806aef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:03:45 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:51746
I20260925 05:03:45.806701 12546 tablet_service.cc:1514] Processing DeleteTablet for tablet ed184fd0cc0f46beba0c750cd1806aef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:03:45 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:47184
I20260925 05:03:45.807370 12411 tablet_service.cc:1514] Processing DeleteTablet for tablet ed184fd0cc0f46beba0c750cd1806aef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:03:45 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:37688
I20260925 05:03:45.807586 12662 tablet_replica.cc:330] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0: stopping tablet replica
I20260925 05:03:45.807976 12662 raft_consensus.cc:2229] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:03:45.808569 12662 raft_consensus.cc:2258] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:03:45.808972 12663 tablet_replica.cc:330] T ed184fd0cc0f46beba0c750cd1806aef P 060ca4e24c53452e833825f270745ac8: stopping tablet replica
I20260925 05:03:45.809340 12663 raft_consensus.cc:2229] T ed184fd0cc0f46beba0c750cd1806aef P 060ca4e24c53452e833825f270745ac8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:03:45.809737 12663 raft_consensus.cc:2258] T ed184fd0cc0f46beba0c750cd1806aef P 060ca4e24c53452e833825f270745ac8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:03:45.814047 12663 ts_tablet_manager.cc:1905] T ed184fd0cc0f46beba0c750cd1806aef P 060ca4e24c53452e833825f270745ac8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:03:45.814932 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 12172
I20260925 05:03:45.820644 12663 ts_tablet_manager.cc:1918] T ed184fd0cc0f46beba0c750cd1806aef P 060ca4e24c53452e833825f270745ac8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:03:45.820801 12663 log.cc:1196] T ed184fd0cc0f46beba0c750cd1806aef P 060ca4e24c53452e833825f270745ac8: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-2/wal/wals/ed184fd0cc0f46beba0c750cd1806aef
I20260925 05:03:45.821237 12663 ts_tablet_manager.cc:1939] T ed184fd0cc0f46beba0c750cd1806aef P 060ca4e24c53452e833825f270745ac8: Deleting consensus metadata
I20260925 05:03:45.821877 12662 ts_tablet_manager.cc:1905] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:03:45.822911 12121 catalog_manager.cc:4841] TS 060ca4e24c53452e833825f270745ac8 (127.30.247.3:37201): tablet ed184fd0cc0f46beba0c750cd1806aef (table test-table [id=e211352458444f5e986bd27d83201880]) successfully deleted
I20260925 05:03:45.823709 12664 tablet_replica.cc:330] T ed184fd0cc0f46beba0c750cd1806aef P c4cf83bae1b44754b6903595609f4ff7: stopping tablet replica
I20260925 05:03:45.824177 12664 raft_consensus.cc:2229] T ed184fd0cc0f46beba0c750cd1806aef P c4cf83bae1b44754b6903595609f4ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:03:45.824841 12664 raft_consensus.cc:2258] T ed184fd0cc0f46beba0c750cd1806aef P c4cf83bae1b44754b6903595609f4ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:03:45.826924 12664 ts_tablet_manager.cc:1905] T ed184fd0cc0f46beba0c750cd1806aef P c4cf83bae1b44754b6903595609f4ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:03:45.830186 12662 ts_tablet_manager.cc:1918] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:03:45.830444 12662 log.cc:1196] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-0/wal/wals/ed184fd0cc0f46beba0c750cd1806aef
I20260925 05:03:45.831039 12662 ts_tablet_manager.cc:1939] T ed184fd0cc0f46beba0c750cd1806aef P 2660d74b41e74d39b19c376a1517e0a0: Deleting consensus metadata
I20260925 05:03:45.832845 12120 catalog_manager.cc:4841] TS 2660d74b41e74d39b19c376a1517e0a0 (127.30.247.1:45597): tablet ed184fd0cc0f46beba0c750cd1806aef (table test-table [id=e211352458444f5e986bd27d83201880]) successfully deleted
I20260925 05:03:45.833710 12664 ts_tablet_manager.cc:1918] T ed184fd0cc0f46beba0c750cd1806aef P c4cf83bae1b44754b6903595609f4ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:03:45.833899 12664 log.cc:1196] T ed184fd0cc0f46beba0c750cd1806aef P c4cf83bae1b44754b6903595609f4ff7: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312430481417-31708-0/minicluster-data/ts-1/wal/wals/ed184fd0cc0f46beba0c750cd1806aef
I20260925 05:03:45.834055 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:45.815704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51764 (local address 127.30.247.1:45597)
0925 05:03:45.816289 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:45.816295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:45.816312 (+ 17us) server_negotiation.cc:407] Connection header received
0925 05:03:45.816363 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:45.816368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:45.816442 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:45.816573 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:45.818431 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.819630 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.821431 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.822101 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.822234 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:45.825898 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:45.825912 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:45.825932 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:45.825974 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:45.830055 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:45.831010 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:45.831018 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:45.831026 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:45.831213 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:45.831641 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:45.831646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:45.831650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:45.832109 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:45.832399 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:45.833375 (+ 976us) server_negotiation.cc:299] Negotiation successful
0925 05:03:45.833614 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459}
I20260925 05:03:45.834463 12664 ts_tablet_manager.cc:1939] T ed184fd0cc0f46beba0c750cd1806aef P c4cf83bae1b44754b6903595609f4ff7: Deleting consensus metadata
I20260925 05:03:45.834717 12327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:03:45.836364 12120 catalog_manager.cc:4841] TS c4cf83bae1b44754b6903595609f4ff7 (127.30.247.2:37937): tablet ed184fd0cc0f46beba0c750cd1806aef (table test-table [id=e211352458444f5e986bd27d83201880]) successfully deleted
I20260925 05:03:45.930648 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 12172
I20260925 05:03:45.947649 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 12336
I20260925 05:03:45.968827 12638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:45.948498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37698 (local address 127.30.247.2:37937)
0925 05:03:45.948684 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:45.948688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:45.948707 (+ 19us) server_negotiation.cc:407] Connection header received
0925 05:03:45.948789 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:45.948793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:45.948853 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:45.948937 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:45.950343 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.951425 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.952836 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:45.953244 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:45.953408 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:45.961425 (+ 8017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:45.961443 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:45.961463 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:45.961507 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:45.965223 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:45.966394 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:45.966405 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:45.966410 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:45.966672 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:45.967096 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:45.967101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:45.967105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:45.967617 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:45.967939 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:45.968226 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 05:03:45.968434 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 05:03:45.969513 12472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:03:46.060578 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 12336
I20260925 05:03:46.077908 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 12481
I20260925 05:03:46.092047 12635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:46.078790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47192 (local address 127.30.247.3:37201)
0925 05:03:46.078957 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:46.078962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:46.078981 (+ 19us) server_negotiation.cc:407] Connection header received
0925 05:03:46.079095 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:46.079098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:46.079161 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:46.079259 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:46.080764 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:46.081485 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:46.083039 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:46.083492 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:46.083593 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:46.086186 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:46.086201 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:46.086216 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:46.086248 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:46.089311 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:46.089955 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:46.089963 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:46.089965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:46.090099 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:46.090506 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:46.090510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:46.090513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:46.090939 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:46.091164 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:46.091553 (+ 389us) server_negotiation.cc:299] Negotiation successful
0925 05:03:46.091704 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 05:03:46.092963 12613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:03:46.179234 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 12481
I20260925 05:03:46.196168 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 12102
I20260925 05:03:46.197402 12165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:03:46.310017 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 12102
I20260925 05:03:46.331672 31708 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 05:03:46.666396 31708 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 05:03:46.666620 31708 mini_ranger.cc:66] Stopping Ranger...
I20260925 05:03:47.011474 31708 mini_ranger.cc:68] Stopped Ranger
2026-09-25 05:03:47.011 UTC [11154] LOG: received smart shutdown request
2026-09-25 05:03:47.015 UTC [11154] LOG: background worker "logical replication launcher" (PID 11163) exited with exit code 1
2026-09-25 05:03:47.015 UTC [11158] LOG: shutting down
2026-09-25 05:03:47.204 UTC [11154] LOG: database system is shut down
2026-09-25T05:03:47Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52503 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:03:47 dist-test-slave-5n86 krb5kdc[12677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:03:47 dist-test-slave-5n86 krb5kdc[12677](info): set up 2 sockets
Sep 25 05:03:47 dist-test-slave-5n86 krb5kdc[12677](info): commencing operation
krb5kdc: starting...
W20260925 05:03:48.449199 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:03:48 dist-test-slave-5n86 krb5kdc[12677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:03:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:03:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:03:48.662159 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:45031
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:34093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:45031,127.30.247.61:34843,127.30.247.60:40731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:03:48.978516 12690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:03:48.978794 12690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:03:48.978881 12690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:03:48.985881 12690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:03:48.985973 12690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:03:48.986018 12690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:03:48.986078 12690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:03:48.986121 12690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:03:48.995383 12690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:34093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:45031,127.30.247.61:34843,127.30.247.60:40731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:45031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:03:48.996651 12690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:03:48.997895 12690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:03:49.009644 12695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:49.009614 12696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:49.010167 12698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:03:49.010648 12690 server_base.cc:1034] running on GCE node
I20260925 05:03:49.011364 12690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:03:49.012789 12690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:03:49.014024 12690 hybrid_clock.cc:648] HybridClock initialized: now 1790312629013984 us; error 64 us; skew 500 ppm
I20260925 05:03:49.014360 12690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:03:49 dist-test-slave-5n86 krb5kdc[12677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312629, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:03:49.017432 12690 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:03:49.018826 12690 webserver.cc:460] Webserver started at http://127.30.247.62:34739/ using document root <none> and password file <none>
I20260925 05:03:49.019464 12690 fs_manager.cc:359] Metadata directory not provided
I20260925 05:03:49.019614 12690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:03:49.019905 12690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:03:49.022854 12690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4"
format_stamp: "Formatted at 2026-09-25 05:03:49 on dist-test-slave-5n86"
server_key: "02759a0046b8a40cc99bea5560b163d8"
server_key_iv: "955e72f605f2dfc02f0a99ef6f703c6c"
server_key_version: "encryptionkey@0"
I20260925 05:03:49.023643 12690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4"
format_stamp: "Formatted at 2026-09-25 05:03:49 on dist-test-slave-5n86"
server_key: "02759a0046b8a40cc99bea5560b163d8"
server_key_iv: "955e72f605f2dfc02f0a99ef6f703c6c"
server_key_version: "encryptionkey@0"
I20260925 05:03:49.028518 12690 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 05:03:49.031931 12705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:49.033179 12690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 05:03:49.033362 12690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4"
format_stamp: "Formatted at 2026-09-25 05:03:49 on dist-test-slave-5n86"
server_key: "02759a0046b8a40cc99bea5560b163d8"
server_key_iv: "955e72f605f2dfc02f0a99ef6f703c6c"
server_key_version: "encryptionkey@0"
I20260925 05:03:49.033555 12690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:03:49.048806 12690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:03:49.052429 12690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:03:49.052724 12690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:03:49.070797 12690 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:45031
I20260925 05:03:49.070848 12756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:45031 every 8 connection(s)
I20260925 05:03:49.072104 12690 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:03:49.075582 12757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:03:49.081364 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 12690
I20260925 05:03:49.081600 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:03:49.082238 31708 external_mini_cluster.cc:1428] Setting key 285fb02a6c928e26e3b1c07f4a9b49f2
I20260925 05:03:49.081954 12757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45031 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:03:49 dist-test-slave-5n86 krb5kdc[12677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312629, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.247.61@KRBTEST.COM; defaulting to no policy
I20260925 05:03:49.103588 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.085042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:45031 (local address 127.0.0.1:55334)
0925 05:03:49.086162 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.086201 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:49.087873 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:49.088708 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:49.088715 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:49.089387 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:03:49.089726 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:49.089743 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:49.091336 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:49.091342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:49.092413 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:49.092425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:49.093328 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:49.093335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:49.093501 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:49.094171 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:03:49.094191 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:03:49.096273 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:03:49.099463 (+ 3190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:49.099475 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:49.099486 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:49.099865 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:49.100267 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:49.100271 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:49.100275 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:49.100425 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:49.101217 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:03:49.101225 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:03:49.101448 (+ 223us) client_negotiation.cc:769] Sending connection context
0925 05:03:49.101693 (+ 245us) client_negotiation.cc:240] Negotiation successful
0925 05:03:49.102312 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"mutex_wait_us":181,"thread_start_us":731,"threads_started":1}
I20260925 05:03:49.103606 12759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.085040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55334 (local address 127.30.247.62:45031)
0925 05:03:49.085863 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:49.085882 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:49.086318 (+ 436us) server_negotiation.cc:407] Connection header received
0925 05:03:49.088225 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:49.088231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:49.088444 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:49.088698 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:49.089909 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:49.091191 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:49.092586 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:49.093109 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:49.093310 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:49.096483 (+ 3173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:49.096491 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:49.096502 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:49.096578 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:49.099340 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:49.099992 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:49.100011 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:49.100013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:49.100151 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:49.100556 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:49.100561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:49.100564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:49.101080 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:49.101274 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:49.101795 (+ 521us) server_negotiation.cc:299] Negotiation successful
0925 05:03:49.102218 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":589,"thread_start_us":384,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.61@KRBTEST.COM" created.
I20260925 05:03:49.106617 12757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:03:49.107925 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.107420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:34843 (local address 127.0.0.1:44282)
0925 05:03:49.107630 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.107765 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 05:03:49.108245 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.247.61:34843: connect: Connection refused (error 111)
I20260925 05:03:49.109982 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.109546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:34843 (local address 127.0.0.1:44288)
0925 05:03:49.109726 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.109819 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 05:03:49.110204 12757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.61:34843: Network error: Client connection negotiation failed: client connection to 127.30.247.61:34843: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.61.keytab.
Entry for principal kudu/127.30.247.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.61.keytab.
I20260925 05:03:49.131546 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.61:34843
--webserver_interface=127.30.247.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.61.keytab
--principal=kudu/127.30.247.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:34093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:45031,127.30.247.61:34843,127.30.247.60:40731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:03:49.159572 12757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 } attempt: 1
I20260925 05:03:49.160727 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.160219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:34843 (local address 127.0.0.1:44294)
0925 05:03:49.160437 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.160577 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 05:03:49.161700 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.161367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:34843 (local address 127.0.0.1:44302)
0925 05:03:49.161513 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.161581 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 05:03:49.161875 12757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.61:34843: Network error: Client connection negotiation failed: client connection to 127.30.247.61:34843: connect: Connection refused (error 111)
I20260925 05:03:49.230134 12757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 } attempt: 2
I20260925 05:03:49.231353 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.230850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:34843 (local address 127.0.0.1:44310)
0925 05:03:49.231112 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.231202 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 05:03:49.232425 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.232070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:34843 (local address 127.0.0.1:44320)
0925 05:03:49.232227 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.232309 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 05:03:49.232646 12757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.61:34843: Network error: Client connection negotiation failed: client connection to 127.30.247.61:34843: connect: Connection refused (error 111)
I20260925 05:03:49.323905 12757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 } attempt: 3
I20260925 05:03:49.325156 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.324690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:34843 (local address 127.0.0.1:44332)
0925 05:03:49.324904 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.325018 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 05:03:49.326130 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.325788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:34843 (local address 127.0.0.1:44334)
0925 05:03:49.325940 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.326008 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 05:03:49.326335 12757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.61:34843: Network error: Client connection negotiation failed: client connection to 127.30.247.61:34843: connect: Connection refused (error 111)
W20260925 05:03:49.447700 12764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:03:49.448019 12764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:03:49.448179 12764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:03:49.455210 12764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:03:49.455350 12764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:03:49.455457 12764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:03:49.455559 12764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:03:49.455632 12764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:03:49.465010 12764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:34093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:45031,127.30.247.61:34843,127.30.247.60:40731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.61.keytab
--principal=kudu/127.30.247.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.61:34843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.247.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:03:49.466228 12764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:03:49.467504 12764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:03:49.469646 12757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 } attempt: 4
I20260925 05:03:49.470902 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.470373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:34843 (local address 127.0.0.1:45130)
0925 05:03:49.470677 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.470768 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260925 05:03:49.471894 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.471533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:34843 (local address 127.0.0.1:45138)
0925 05:03:49.471694 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.471780 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 05:03:49.472072 12757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.61:34843: Network error: Client connection negotiation failed: client connection to 127.30.247.61:34843: connect: Connection refused (error 111)
W20260925 05:03:49.479223 12769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:49.479223 12770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:49.479291 12772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:03:49.480020 12764 server_base.cc:1034] running on GCE node
I20260925 05:03:49.480801 12764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:03:49.482134 12764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:03:49.483361 12764 hybrid_clock.cc:648] HybridClock initialized: now 1790312629483323 us; error 65 us; skew 500 ppm
I20260925 05:03:49.483697 12764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:03:49 dist-test-slave-5n86 krb5kdc[12677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312629, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:03:49.494452 12764 init.cc:377] Logged in from keytab as kudu/127.30.247.61@KRBTEST.COM (short username kudu)
I20260925 05:03:49.495860 12764 webserver.cc:460] Webserver started at http://127.30.247.61:44999/ using document root <none> and password file <none>
I20260925 05:03:49.496523 12764 fs_manager.cc:359] Metadata directory not provided
I20260925 05:03:49.496726 12764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:03:49.497022 12764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:03:49.500025 12764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-1/data/instance:
uuid: "db73e26a10314db2963941a9662e8b74"
format_stamp: "Formatted at 2026-09-25 05:03:49 on dist-test-slave-5n86"
server_key: "d9bc2b9c1c255ce24560ac3e204e2ca3"
server_key_iv: "0579211639c58f31d878ba5d727a3db3"
server_key_version: "encryptionkey@0"
I20260925 05:03:49.500869 12764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-1/wal/instance:
uuid: "db73e26a10314db2963941a9662e8b74"
format_stamp: "Formatted at 2026-09-25 05:03:49 on dist-test-slave-5n86"
server_key: "d9bc2b9c1c255ce24560ac3e204e2ca3"
server_key_iv: "0579211639c58f31d878ba5d727a3db3"
server_key_version: "encryptionkey@0"
I20260925 05:03:49.505702 12764 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 05:03:49.508993 12779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:49.510232 12764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 05:03:49.510412 12764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-1/wal
uuid: "db73e26a10314db2963941a9662e8b74"
format_stamp: "Formatted at 2026-09-25 05:03:49 on dist-test-slave-5n86"
server_key: "d9bc2b9c1c255ce24560ac3e204e2ca3"
server_key_iv: "0579211639c58f31d878ba5d727a3db3"
server_key_version: "encryptionkey@0"
I20260925 05:03:49.510598 12764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:03:49.528000 12764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:03:49.531881 12764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:03:49.532155 12764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:03:49.549854 12764 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.61:34843
I20260925 05:03:49.549896 12830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.61:34843 every 8 connection(s)
I20260925 05:03:49.551164 12764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-1/data/info.pb
I20260925 05:03:49.554997 12831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:03:49.560359 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 12764
I20260925 05:03:49.560590 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-1/wal/instance
I20260925 05:03:49.561055 31708 external_mini_cluster.cc:1428] Setting key f39601b6360f76c86f4a86140a640689
I20260925 05:03:49.561280 12831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45031 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:03:49 dist-test-slave-5n86 krb5kdc[12677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312629, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.61@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.247.60@KRBTEST.COM; defaulting to no policy
I20260925 05:03:49.583050 12759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.563746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36506 (local address 127.30.247.62:45031)
0925 05:03:49.563918 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:49.563922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:49.565259 (+ 1337us) server_negotiation.cc:407] Connection header received
0925 05:03:49.567088 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:49.567094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:49.567187 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:49.567301 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:49.569394 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:49.570066 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:49.572521 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:49.573062 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:49.573183 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:49.577307 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:49.577312 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:49.577316 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:49.577347 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:49.580056 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:49.580867 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:49.580871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:49.580874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:49.580991 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:49.581413 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:49.581418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:49.581420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:49.581689 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:49.581853 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:49.582511 (+ 658us) server_negotiation.cc:299] Negotiation successful
0925 05:03:49.582725 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 05:03:49.583936 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.564312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:45031 (local address 127.0.0.1:36506)
0925 05:03:49.565101 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.565143 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:49.566819 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:49.567560 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:49.567575 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:49.568337 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:03:49.569167 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:49.569193 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:49.570232 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:49.570240 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:49.572358 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:49.572375 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:49.573291 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:49.573300 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:49.573626 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:49.574391 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:03:49.574422 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:03:49.577077 (+ 2655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:03:49.580214 (+ 3137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:49.580223 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:49.580237 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:49.580727 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:49.581099 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:49.581104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:49.581108 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:49.581280 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:49.581816 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:03:49.581823 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:03:49.582145 (+ 322us) client_negotiation.cc:769] Sending connection context
0925 05:03:49.582398 (+ 253us) client_negotiation.cc:240] Negotiation successful
0925 05:03:49.582858 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":375,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.60@KRBTEST.COM" created.
I20260925 05:03:49.585757 12831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:03:49 dist-test-slave-5n86 krb5kdc[12677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312629, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.61@KRBTEST.COM for kudu/127.30.247.61@KRBTEST.COM
I20260925 05:03:49.600891 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.586568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45154 (local address 127.30.247.61:34843)
0925 05:03:49.587192 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:49.587203 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:49.587235 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:03:49.587290 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:49.587301 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:49.587534 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:49.587689 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:49.588290 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:49.590038 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:49.591135 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:49.591724 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:49.591820 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:49.595455 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:49.595466 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:49.595477 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:49.595503 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:49.598155 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:49.598758 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:49.598765 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:49.598768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:49.598877 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:49.599238 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:49.599246 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:49.599249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:49.599570 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:49.599805 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:49.600212 (+ 407us) server_negotiation.cc:299] Negotiation successful
0925 05:03:49.600512 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":112,"server-negotiator.queue_time_us":436,"thread_start_us":264,"threads_started":1}
I20260925 05:03:49.600970 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.586541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:34843 (local address 127.0.0.1:45154)
0925 05:03:49.586749 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.586777 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:49.586902 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:49.587684 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:49.587688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:49.588018 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:03:49.588141 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:49.588148 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:49.590169 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:49.590172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:49.590997 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:49.591008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:49.591876 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:49.591883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:49.592182 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:49.592833 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:03:49.592859 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:03:49.595245 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:03:49.598293 (+ 3048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:49.598300 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:49.598303 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:49.598637 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:49.599004 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:49.599008 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:49.599012 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:49.599133 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:49.599738 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:03:49.599747 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:03:49.599925 (+ 178us) client_negotiation.cc:769] Sending connection context
0925 05:03:49.600086 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 05:03:49.600389 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 05:03:49.603550 12831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:03:49.604622 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.604136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:40731 (local address 127.0.0.1:54688)
0925 05:03:49.604295 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.604420 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 05:03:49.604863 12782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
I20260925 05:03:49.606338 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.605999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:40731 (local address 127.0.0.1:54702)
0925 05:03:49.606175 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.606243 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 05:03:49.606537 12831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.60:40731: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.60.keytab.
Entry for principal kudu/127.30.247.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.60.keytab.
I20260925 05:03:49.611855 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.60:40731
--webserver_interface=127.30.247.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.60.keytab
--principal=kudu/127.30.247.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:34093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:45031,127.30.247.61:34843,127.30.247.60:40731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:03:49.655858 12831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } attempt: 1
I20260925 05:03:49.657233 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.656746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:40731 (local address 127.0.0.1:54712)
0925 05:03:49.656978 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.657080 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 05:03:49.658380 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.658004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:40731 (local address 127.0.0.1:54724)
0925 05:03:49.658192 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.658265 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260925 05:03:49.658655 12831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.60:40731: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
I20260925 05:03:49.726909 12831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } attempt: 2
I20260925 05:03:49.728091 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.727558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:40731 (local address 127.0.0.1:54740)
0925 05:03:49.727787 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.727903 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 05:03:49.729218 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.728842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:40731 (local address 127.0.0.1:54756)
0925 05:03:49.728995 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.729081 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 05:03:49.729461 12831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.60:40731: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
I20260925 05:03:49.771317 12757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 } attempt: 5
Sep 25 05:03:49 dist-test-slave-5n86 krb5kdc[12677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312629, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for kudu/127.30.247.61@KRBTEST.COM
I20260925 05:03:49.784324 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.772319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45160 (local address 127.30.247.61:34843)
0925 05:03:49.772567 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:49.772573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:49.772598 (+ 25us) server_negotiation.cc:407] Connection header received
0925 05:03:49.772665 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:49.772669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:49.772726 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:49.772835 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:49.773656 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:49.774333 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:49.775515 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:49.775979 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:49.776114 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:49.779305 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:49.779310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:49.779313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:49.779344 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:49.781857 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:49.782476 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:49.782484 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:49.782488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:49.782606 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:49.782964 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:49.782970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:49.782974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:49.783242 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:49.783395 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:49.783774 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 05:03:49.783991 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 05:03:49.784323 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.772058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:34843 (local address 127.0.0.1:45160)
0925 05:03:49.772255 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.772280 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:49.772416 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:49.772885 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:49.772892 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:49.773283 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:03:49.773478 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:49.773487 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:49.774474 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:49.774480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:49.775351 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:49.775382 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:49.776128 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:49.776134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:49.776319 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:49.776958 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:03:49.776976 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:03:49.779117 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:03:49.781998 (+ 2881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:49.782005 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:49.782009 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:49.782349 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:49.782716 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:49.782722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:49.782725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:49.782858 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:49.783359 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:03:49.783363 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:03:49.783525 (+ 162us) client_negotiation.cc:769] Sending connection context
0925 05:03:49.783667 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 05:03:49.783963 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 05:03:49.785509 12757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:03:49.786419 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.786067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:40731 (local address 127.0.0.1:54764)
0925 05:03:49.786216 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.786291 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 05:03:49.787274 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.786974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:40731 (local address 127.0.0.1:54772)
0925 05:03:49.787109 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.787190 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 05:03:49.787488 12757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.60:40731: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
I20260925 05:03:49.820668 12831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } attempt: 3
I20260925 05:03:49.821884 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.821280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:40731 (local address 127.0.0.1:54788)
0925 05:03:49.821537 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.821659 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 05:03:49.823156 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.822640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:40731 (local address 127.0.0.1:54798)
0925 05:03:49.822889 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.822991 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260925 05:03:49.823410 12831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.60:40731: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
I20260925 05:03:49.838709 12757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } attempt: 1
I20260925 05:03:49.839864 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.839316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:40731 (local address 127.0.0.1:54804)
0925 05:03:49.839544 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.839663 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 05:03:49.841007 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.840593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:40731 (local address 127.0.0.1:54818)
0925 05:03:49.840763 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.840859 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 05:03:49.841279 12757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.60:40731: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
I20260925 05:03:49.909492 12757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } attempt: 2
I20260925 05:03:49.910717 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.910119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:40731 (local address 127.0.0.1:54834)
0925 05:03:49.910381 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.910512 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 05:03:49.911850 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.911436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:40731 (local address 127.0.0.1:54840)
0925 05:03:49.911642 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.911725 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260925 05:03:49.912070 12757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.60:40731: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
W20260925 05:03:49.928272 12838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:03:49.928524 12838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:03:49.928753 12838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:03:49.935631 12838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:03:49.935720 12838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:03:49.935834 12838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:03:49.935973 12838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:03:49.936038 12838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:03:49.945626 12838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:34093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:45031,127.30.247.61:34843,127.30.247.60:40731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.60.keytab
--principal=kudu/127.30.247.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.60:40731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.247.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:03:49.946867 12838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:03:49.948123 12838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:03:49.960078 12846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:03:49.966161 12838 server_base.cc:1034] running on GCE node
I20260925 05:03:49.966641 12831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } attempt: 4
W20260925 05:03:49.967798 12843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:49.967862 12844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:03:49.968428 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.967756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:40731 (local address 127.0.0.1:54854)
0925 05:03:49.968122 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.968225 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260925 05:03:49.969295 12838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:03:49.969609 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:49.969211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:40731 (local address 127.0.0.1:54866)
0925 05:03:49.969400 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:49.969488 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260925 05:03:49.969923 12831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.60:40731: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
I20260925 05:03:49.970925 12838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:03:49.972160 12838 hybrid_clock.cc:648] HybridClock initialized: now 1790312629972123 us; error 37 us; skew 500 ppm
I20260925 05:03:49.972625 12838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:03:49 dist-test-slave-5n86 krb5kdc[12677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312629, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:03:49.975657 12838 init.cc:377] Logged in from keytab as kudu/127.30.247.60@KRBTEST.COM (short username kudu)
I20260925 05:03:49.977102 12838 webserver.cc:460] Webserver started at http://127.30.247.60:44049/ using document root <none> and password file <none>
I20260925 05:03:49.977731 12838 fs_manager.cc:359] Metadata directory not provided
I20260925 05:03:49.977823 12838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:03:49.978055 12838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:03:49.981191 12838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-2/data/instance:
uuid: "ead8c59838dc4a04b8a4233c2470d516"
format_stamp: "Formatted at 2026-09-25 05:03:49 on dist-test-slave-5n86"
server_key: "757c1fba0a7bb3c23eb53700cd6f756d"
server_key_iv: "79cb7d5643d3e8ac4d7a18b5a6a92521"
server_key_version: "encryptionkey@0"
I20260925 05:03:49.981913 12838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-2/wal/instance:
uuid: "ead8c59838dc4a04b8a4233c2470d516"
format_stamp: "Formatted at 2026-09-25 05:03:49 on dist-test-slave-5n86"
server_key: "757c1fba0a7bb3c23eb53700cd6f756d"
server_key_iv: "79cb7d5643d3e8ac4d7a18b5a6a92521"
server_key_version: "encryptionkey@0"
I20260925 05:03:49.986783 12838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:03:49.990056 12853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:49.991397 12838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:03:49.991582 12838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-2/wal
uuid: "ead8c59838dc4a04b8a4233c2470d516"
format_stamp: "Formatted at 2026-09-25 05:03:49 on dist-test-slave-5n86"
server_key: "757c1fba0a7bb3c23eb53700cd6f756d"
server_key_iv: "79cb7d5643d3e8ac4d7a18b5a6a92521"
server_key_version: "encryptionkey@0"
I20260925 05:03:49.991753 12838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:03:50.013558 12838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:03:50.017827 12838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:03:50.018052 12838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:03:50.018306 12757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } attempt: 3
I20260925 05:03:50.019431 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:50.018911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:40731 (local address 127.0.0.1:54870)
0925 05:03:50.019145 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:50.019249 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260925 05:03:50.020684 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:50.020248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:40731 (local address 127.0.0.1:54878)
0925 05:03:50.020457 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:50.020527 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 05:03:50.020974 12757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.60:40731: Network error: Client connection negotiation failed: client connection to 127.30.247.60:40731: connect: Connection refused (error 111)
I20260925 05:03:50.036808 12838 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.60:40731
I20260925 05:03:50.036933 12904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.60:40731 every 8 connection(s)
I20260925 05:03:50.038298 12838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-2/data/info.pb
I20260925 05:03:50.040943 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 12838
I20260925 05:03:50.041146 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312430481417-31708-0/minicluster-data/master-2/wal/instance
I20260925 05:03:50.041603 31708 external_mini_cluster.cc:1428] Setting key 5f563590205199e8149f1d2ae7455f47
I20260925 05:03:50.042047 12905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:03:50.051038 12905 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45031 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:03:50 dist-test-slave-5n86 krb5kdc[12677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.60@KRBTEST.COM
Sep 25 05:03:50 dist-test-slave-5n86 krb5kdc[12677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312629, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.60@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:03:50.064023 12908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:50.043375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54890 (local address 127.30.247.60:40731)
0925 05:03:50.044179 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:50.044193 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:50.044239 (+ 46us) server_negotiation.cc:407] Connection header received
0925 05:03:50.046697 (+ 2458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:50.046725 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:50.047217 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:50.047622 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:50.049067 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.050338 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.051814 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.052442 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.052666 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:50.057123 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:50.057130 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:50.057139 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:50.057177 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:50.060178 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:50.060952 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:50.060967 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:50.060972 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:50.061139 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:50.061549 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:50.061553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:50.061554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:50.062020 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:50.062320 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:50.062768 (+ 448us) server_negotiation.cc:299] Negotiation successful
0925 05:03:50.063141 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":342,"threads_started":1}
I20260925 05:03:50.067222 12909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:50.053367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:45031 (local address 127.0.0.1:36508)
0925 05:03:50.054160 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:50.054209 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:50.054370 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:50.054788 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:50.054793 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:50.055335 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:03:50.055713 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:50.055725 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.056806 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.056812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:50.057837 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:50.057848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.058617 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.058623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:50.058788 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:50.059339 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:03:50.059355 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:03:50.061433 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:03:50.064558 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:50.064569 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:50.064575 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:50.064965 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:50.065385 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:50.065391 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:50.065395 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:50.065552 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:50.066186 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:03:50.066191 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:03:50.066317 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 05:03:50.066511 (+ 194us) client_negotiation.cc:240] Negotiation successful
0925 05:03:50.066793 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":455,"threads_started":1}
I20260925 05:03:50.067438 12759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:50.053382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36508 (local address 127.30.247.62:45031)
0925 05:03:50.053601 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:50.053611 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:50.054322 (+ 711us) server_negotiation.cc:407] Connection header received
0925 05:03:50.054510 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:50.054518 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:50.054611 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:50.054727 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:50.055871 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.056662 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.057991 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.058419 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.058573 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:50.061628 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:50.061633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:50.061636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:50.061666 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:50.064400 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:50.065084 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:50.065091 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:50.065098 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:50.065257 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:50.065662 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:50.065667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:50.065671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:50.065912 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:50.066086 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:50.066625 (+ 539us) server_negotiation.cc:299] Negotiation successful
0925 05:03:50.066837 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 05:03:50.068830 12905 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:03:50 dist-test-slave-5n86 krb5kdc[12677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312629, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.60@KRBTEST.COM for kudu/127.30.247.61@KRBTEST.COM
Sep 25 05:03:50 dist-test-slave-5n86 krb5kdc[12677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:03:50.083225 12909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:50.069445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:34843 (local address 127.0.0.1:45162)
0925 05:03:50.069596 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:50.069615 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:50.069702 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:50.070016 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:50.070020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:50.070469 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:03:50.070597 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:50.070605 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.071730 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.071737 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:50.072947 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:50.072962 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.073917 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.073921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:50.074173 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:50.074736 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:03:50.074752 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:03:50.076805 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:03:50.080661 (+ 3856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:50.080667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:50.080671 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:50.081111 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:50.081491 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:50.081497 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:50.081501 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:50.081650 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:50.082264 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:03:50.082269 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:03:50.082451 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 05:03:50.082593 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 05:03:50.082872 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 05:03:50.083257 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:50.069551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45162 (local address 127.30.247.61:34843)
0925 05:03:50.069732 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:50.069741 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:50.069760 (+ 19us) server_negotiation.cc:407] Connection header received
0925 05:03:50.069816 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:50.069822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:50.069894 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:50.069998 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:50.070744 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.071596 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.073103 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.073617 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.073747 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:50.077003 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:50.077010 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:50.077014 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:50.077053 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:50.080463 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:50.081231 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:50.081238 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:50.081241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:50.081381 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:50.081762 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:50.081768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:50.081772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:50.082151 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:50.082310 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:50.082686 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 05:03:50.082903 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 05:03:50.083803 12759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:50.066754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36524 (local address 127.30.247.62:45031)
0925 05:03:50.067679 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:50.067685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:50.067701 (+ 16us) server_negotiation.cc:407] Connection header received
0925 05:03:50.067752 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:50.067760 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:50.067839 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:50.067942 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:50.069394 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.070316 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.072196 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.072864 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.072994 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:50.077682 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:50.077688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:50.077693 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:50.077740 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:50.080740 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:50.081598 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:50.081602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:50.081604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:50.081730 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:50.082089 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:50.082092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:50.082094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:50.082335 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:50.082640 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:50.082921 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 05:03:50.083127 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":817,"thread_start_us":780,"threads_started":1}
I20260925 05:03:50.084178 12905 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:03:50 dist-test-slave-5n86 krb5kdc[12677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312629, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.60@KRBTEST.COM for kudu/127.30.247.60@KRBTEST.COM
Sep 25 05:03:50 dist-test-slave-5n86 krb5kdc[12677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.61@KRBTEST.COM
I20260925 05:03:50.096670 12909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:50.084956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:40731 (local address 127.0.0.1:54902)
0925 05:03:50.085148 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:50.085167 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:50.085259 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:50.085550 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:50.085554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:50.085919 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:03:50.086037 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:50.086045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.086944 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.086952 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:50.087776 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:50.087786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.088478 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.088492 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:50.088680 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:50.089236 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:03:50.089252 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:03:50.091260 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:03:50.094163 (+ 2903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:50.094168 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:50.094171 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:50.094500 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:50.094951 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:50.094954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:50.094957 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:50.095079 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:50.095724 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:03:50.095730 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:03:50.095920 (+ 190us) client_negotiation.cc:769] Sending connection context
0925 05:03:50.096034 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 05:03:50.096226 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 05:03:50.096825 12908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:50.085114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54902 (local address 127.30.247.60:40731)
0925 05:03:50.085269 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:50.085277 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:50.085297 (+ 20us) server_negotiation.cc:407] Connection header received
0925 05:03:50.085372 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:50.085376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:50.085427 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:50.085536 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:50.086166 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.086821 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.087910 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.088323 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.088457 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:50.091415 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:50.091421 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:50.091426 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:50.091463 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:50.094030 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:50.094653 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:50.094659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:50.094662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:50.094842 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:50.095186 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:50.095191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:50.095194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:50.095602 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:50.095759 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:50.096155 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 05:03:50.096418 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 05:03:50.100207 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:50.085742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45168 (local address 127.30.247.61:34843)
0925 05:03:50.085905 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:50.085912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:50.085928 (+ 16us) server_negotiation.cc:407] Connection header received
0925 05:03:50.086079 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:50.086084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:50.086149 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:50.086244 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:50.087434 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.088133 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.089446 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.089865 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.089993 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:50.094050 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:50.094056 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:50.094060 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:50.094094 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:50.097403 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:50.098248 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:50.098252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:50.098255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:50.098443 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:50.098844 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:50.098849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:50.098853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:50.099119 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:50.099414 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:50.099686 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 05:03:50.099903 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 05:03:50.102356 12905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ead8c59838dc4a04b8a4233c2470d516: Bootstrap starting.
I20260925 05:03:50.105041 31708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:03:50.105970 12905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ead8c59838dc4a04b8a4233c2470d516: Neither blocks nor log segments found. Creating new log.
I20260925 05:03:50.106962 12905 log.cc:824] T 00000000000000000000000000000000 P ead8c59838dc4a04b8a4233c2470d516: Log is configured to *not* fsync() on all Append() calls
I20260925 05:03:50.109987 12905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ead8c59838dc4a04b8a4233c2470d516: No bootstrap required, opened a new log
I20260925 05:03:50.123108 12905 raft_consensus.cc:348] T 00000000000000000000000000000000 P ead8c59838dc4a04b8a4233c2470d516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45031 } } peers { permanent_uuid: "db73e26a10314db2963941a9662e8b74" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 } } peers { permanent_uuid: "ead8c59838dc4a04b8a4233c2470d516" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } }
I20260925 05:03:50.123534 12905 raft_consensus.cc:374] T 00000000000000000000000000000000 P ead8c59838dc4a04b8a4233c2470d516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:03:50.123658 12905 raft_consensus.cc:729] T 00000000000000000000000000000000 P ead8c59838dc4a04b8a4233c2470d516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ead8c59838dc4a04b8a4233c2470d516, State: Initialized, Role: FOLLOWER
I20260925 05:03:50.124315 12905 consensus_queue.cc:260] T 00000000000000000000000000000000 P ead8c59838dc4a04b8a4233c2470d516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45031 } } peers { permanent_uuid: "db73e26a10314db2963941a9662e8b74" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 } } peers { permanent_uuid: "ead8c59838dc4a04b8a4233c2470d516" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } }
I20260925 05:03:50.127576 12919 sys_catalog.cc:455] T 00000000000000000000000000000000 P ead8c59838dc4a04b8a4233c2470d516 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45031 } } peers { permanent_uuid: "db73e26a10314db2963941a9662e8b74" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 } } peers { permanent_uuid: "ead8c59838dc4a04b8a4233c2470d516" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } } }
I20260925 05:03:50.128034 12919 sys_catalog.cc:458] T 00000000000000000000000000000000 P ead8c59838dc4a04b8a4233c2470d516 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:03:50.128942 12905 sys_catalog.cc:564] T 00000000000000000000000000000000 P ead8c59838dc4a04b8a4233c2470d516 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:03:50.136194 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:50.112003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45170 (local address 127.30.247.61:34843)
0925 05:03:50.112162 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:50.112169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:50.112441 (+ 272us) server_negotiation.cc:407] Connection header received
0925 05:03:50.113210 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:50.113221 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:50.113330 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:50.113438 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:50.116730 (+ 3292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.118200 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.120983 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.121880 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.122000 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:50.127239 (+ 5239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:50.127246 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:50.127251 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:50.127292 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:50.131216 (+ 3924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:50.132362 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:50.132370 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:50.132377 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:50.132580 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:50.133149 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:50.133157 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:50.133163 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:50.133635 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:50.135208 (+ 1573us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:50.135374 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 05:03:50.135663 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 05:03:50.143649 12759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:50.111511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36528 (local address 127.30.247.62:45031)
0925 05:03:50.111692 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:50.111698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:50.123601 (+ 11903us) server_negotiation.cc:407] Connection header received
0925 05:03:50.123891 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:50.123901 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:50.123993 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:50.124075 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:50.128277 (+ 4202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.129406 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.131668 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.132368 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.132476 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:50.136099 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:50.136107 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:50.136112 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:50.136157 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:50.140512 (+ 4355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:50.141370 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:50.141375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:50.141379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:50.141522 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:50.142012 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:50.142018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:50.142021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:50.142397 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:50.142650 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:50.143066 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 05:03:50.143292 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 05:03:50.147444 12930 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ead8c59838dc4a04b8a4233c2470d516: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:03:50.147624 12930 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:03:50.148679 12908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:50.112931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54914 (local address 127.30.247.60:40731)
0925 05:03:50.113174 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:50.113181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:50.127151 (+ 13970us) server_negotiation.cc:407] Connection header received
0925 05:03:50.127392 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:50.127397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:50.127492 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:50.127570 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:50.129240 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.130370 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.132588 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.133343 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.133468 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:50.138162 (+ 4694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:50.138168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:50.138172 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:50.138212 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:50.142455 (+ 4243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:50.145522 (+ 3067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:50.145526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:50.145529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:50.145693 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:50.146178 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:50.146183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:50.146187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:50.146612 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:50.147867 (+ 1255us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:50.148000 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 05:03:50.148228 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 05:03:50.145455 12838 master_runner.cc:186] Error getting master registration for 127.30.247.62:45031: OK, instance_id { permanent_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" instance_seqno: 1790312629065660 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 05:03:50.155206 12838 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 05:03:50.198232 12757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } attempt: 4
Sep 25 05:03:50 dist-test-slave-5n86 krb5kdc[12677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312629, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for kudu/127.30.247.60@KRBTEST.COM
I20260925 05:03:50.212939 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:50.198831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:40731 (local address 127.0.0.1:54916)
0925 05:03:50.199024 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:50.199046 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:50.199168 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:50.199544 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:50.199548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:50.199907 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:03:50.200070 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:50.200078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.201533 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.201537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:50.202457 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:50.202468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.203464 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.203469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:50.203615 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:50.204262 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:03:50.204288 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:03:50.206509 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:03:50.210704 (+ 4195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:50.210707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:50.210710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:50.211058 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:50.211445 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:50.211448 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:50.211451 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:50.211588 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:50.212169 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:03:50.212172 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:03:50.212294 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 05:03:50.212412 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 05:03:50.212669 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 05:03:50.213200 12908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:50.199053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54916 (local address 127.30.247.60:40731)
0925 05:03:50.199252 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:50.199259 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:50.199276 (+ 17us) server_negotiation.cc:407] Connection header received
0925 05:03:50.199326 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:50.199331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:50.199407 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:50.199512 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:50.200208 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.201402 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.202597 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.203306 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.203448 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:50.206688 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:50.206694 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:50.206700 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:50.206742 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:50.210575 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:50.211171 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:50.211177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:50.211181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:50.211337 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:50.211690 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:50.211695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:50.211699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:50.212050 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:50.212196 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:50.212566 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 05:03:50.212808 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 05:03:50.217722 12757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4: Bootstrap starting.
I20260925 05:03:50.221716 12757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4: Neither blocks nor log segments found. Creating new log.
I20260925 05:03:50.222725 12757 log.cc:824] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4: Log is configured to *not* fsync() on all Append() calls
I20260925 05:03:50.225414 12757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4: No bootstrap required, opened a new log
I20260925 05:03:50.232898 12757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45031 } } peers { permanent_uuid: "db73e26a10314db2963941a9662e8b74" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 } } peers { permanent_uuid: "ead8c59838dc4a04b8a4233c2470d516" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } }
I20260925 05:03:50.233173 12757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:03:50.233330 12757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e6b9bf4a0744d6c9e9408e746d7f7b4, State: Initialized, Role: FOLLOWER
I20260925 05:03:50.233925 12757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45031 } } peers { permanent_uuid: "db73e26a10314db2963941a9662e8b74" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 } } peers { permanent_uuid: "ead8c59838dc4a04b8a4233c2470d516" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } }
I20260925 05:03:50.237185 12933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45031 } } peers { permanent_uuid: "db73e26a10314db2963941a9662e8b74" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 } } peers { permanent_uuid: "ead8c59838dc4a04b8a4233c2470d516" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } } }
I20260925 05:03:50.237592 12933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:03:50.238265 12757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 05:03:50.247942 12944 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:03:50.248132 12944 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:03:50.247069 12690 master_runner.cc:186] Error getting master registration for 127.30.247.61:34843: OK, instance_id { permanent_uuid: "db73e26a10314db2963941a9662e8b74" instance_seqno: 1790312629544829 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 05:03:50.255194 12690 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 05:03:50.268370 12933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:03:50.268689 12933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45031 } } peers { permanent_uuid: "db73e26a10314db2963941a9662e8b74" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 } } peers { permanent_uuid: "ead8c59838dc4a04b8a4233c2470d516" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } }
I20260925 05:03:50.269146 12831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } attempt: 5
I20260925 05:03:50.270236 12933 leader_election.cc:288] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db73e26a10314db2963941a9662e8b74 (127.30.247.61:34843), ead8c59838dc4a04b8a4233c2470d516 (127.30.247.60:40731)
W20260925 05:03:50.270670 12859 tablet.cc:2369] T 00000000000000000000000000000000 P ead8c59838dc4a04b8a4233c2470d516: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:03:50.272460 12880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ead8c59838dc4a04b8a4233c2470d516" is_pre_election: true
I20260925 05:03:50.273185 12880 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ead8c59838dc4a04b8a4233c2470d516 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e6b9bf4a0744d6c9e9408e746d7f7b4 in term 0.
I20260925 05:03:50.273837 12708 leader_election.cc:302] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e6b9bf4a0744d6c9e9408e746d7f7b4, ead8c59838dc4a04b8a4233c2470d516; no voters:
I20260925 05:03:50.274230 12933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:03:50.274402 12933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:03:50.274513 12933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:03:50.277765 12933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45031 } } peers { permanent_uuid: "db73e26a10314db2963941a9662e8b74" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 } } peers { permanent_uuid: "ead8c59838dc4a04b8a4233c2470d516" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } }
I20260925 05:03:50.278975 12880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ead8c59838dc4a04b8a4233c2470d516"
I20260925 05:03:50.279188 12880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ead8c59838dc4a04b8a4233c2470d516 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:03:50.282459 12880 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ead8c59838dc4a04b8a4233c2470d516 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e6b9bf4a0744d6c9e9408e746d7f7b4 in term 1.
I20260925 05:03:50.282836 12708 leader_election.cc:302] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e6b9bf4a0744d6c9e9408e746d7f7b4, ead8c59838dc4a04b8a4233c2470d516; no voters:
I20260925 05:03:50.284898 12933 leader_election.cc:288] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [CANDIDATE]: Term 1 election: Requested vote from peers db73e26a10314db2963941a9662e8b74 (127.30.247.61:34843), ead8c59838dc4a04b8a4233c2470d516 (127.30.247.60:40731)
I20260925 05:03:50.285094 12933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:03:50.285238 12933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [term 1 LEADER]: Becoming Leader. State: Replica: 7e6b9bf4a0744d6c9e9408e746d7f7b4, State: Running, Role: LEADER
I20260925 05:03:50.285801 12933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45031 } } peers { permanent_uuid: "db73e26a10314db2963941a9662e8b74" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 } } peers { permanent_uuid: "ead8c59838dc4a04b8a4233c2470d516" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } }
Sep 25 05:03:50 dist-test-slave-5n86 krb5kdc[12677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312629, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.61@KRBTEST.COM for kudu/127.30.247.60@KRBTEST.COM
I20260925 05:03:50.302618 12933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e6b9bf4a0744d6c9e9408e746d7f7b4. Latest consensus state: current_term: 1 leader_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45031 } } peers { permanent_uuid: "db73e26a10314db2963941a9662e8b74" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 } } peers { permanent_uuid: "ead8c59838dc4a04b8a4233c2470d516" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } } }
I20260925 05:03:50.302944 12933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [sys.catalog]: This master's current role is: LEADER
W20260925 05:03:50.305253 12711 tablet.cc:2369] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:03:50.310092 12948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:03:50.310640 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:50.269651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:40731 (local address 127.0.0.1:54930)
0925 05:03:50.269978 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:50.269998 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:50.270102 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:50.271960 (+ 1858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:50.271966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:50.272509 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:03:50.272782 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:50.272797 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.274497 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.274504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:50.275877 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:50.275892 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.277264 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.277271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:50.277497 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:50.278342 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:03:50.278753 (+ 411us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:03:50.288490 (+ 9737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:03:50.296505 (+ 8015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:50.296514 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:50.296521 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:50.297928 (+ 1407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:50.302315 (+ 4387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:50.302321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:50.302354 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:50.302640 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:50.308238 (+ 5598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:03:50.308246 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:03:50.308741 (+ 495us) client_negotiation.cc:769] Sending connection context
0925 05:03:50.309153 (+ 412us) client_negotiation.cc:240] Negotiation successful
0925 05:03:50.309810 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200}
I20260925 05:03:50.312574 12948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:03:50.317378 12908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:50.270345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54930 (local address 127.30.247.60:40731)
0925 05:03:50.271431 (+ 1086us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:50.271436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:50.271456 (+ 20us) server_negotiation.cc:407] Connection header received
0925 05:03:50.271514 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:50.271521 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:50.271601 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:50.271697 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:50.273033 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.274143 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.276138 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:50.276917 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:50.277034 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:50.288892 (+ 11858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:50.288901 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:50.288912 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:50.288947 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:50.296133 (+ 7186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:50.298307 (+ 2174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:50.298355 (+ 48us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:50.298360 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:50.301836 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:50.303537 (+ 1701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:50.303548 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:50.303678 (+ 130us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:50.307461 (+ 3783us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:50.311785 (+ 4324us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:50.312929 (+ 1144us) server_negotiation.cc:299] Negotiation successful
0925 05:03:50.314770 (+ 1841us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":960}
I20260925 05:03:50.325085 12831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db73e26a10314db2963941a9662e8b74: Bootstrap starting.
I20260925 05:03:50.333439 12831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db73e26a10314db2963941a9662e8b74: Neither blocks nor log segments found. Creating new log.
I20260925 05:03:50.334504 12880 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ead8c59838dc4a04b8a4233c2470d516 [term 1 FOLLOWER]: Refusing update from remote peer 7e6b9bf4a0744d6c9e9408e746d7f7b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:03:50.335532 12946 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ead8c59838dc4a04b8a4233c2470d516" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:03:50.337018 12831 log.cc:824] T 00000000000000000000000000000000 P db73e26a10314db2963941a9662e8b74: Log is configured to *not* fsync() on all Append() calls
I20260925 05:03:50.351590 12831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db73e26a10314db2963941a9662e8b74: No bootstrap required, opened a new log
I20260925 05:03:50.355873 12919 sys_catalog.cc:455] T 00000000000000000000000000000000 P ead8c59838dc4a04b8a4233c2470d516 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e6b9bf4a0744d6c9e9408e746d7f7b4. Latest consensus state: current_term: 1 leader_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45031 } } peers { permanent_uuid: "db73e26a10314db2963941a9662e8b74" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 } } peers { permanent_uuid: "ead8c59838dc4a04b8a4233c2470d516" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } } }
I20260925 05:03:50.356276 12919 sys_catalog.cc:458] T 00000000000000000000000000000000 P ead8c59838dc4a04b8a4233c2470d516 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:03:50.360469 12949 mvcc.cc:205] Tried to move back new op lower bound from 7333120533837295616 to 7333120533663551488. Current Snapshot: MvccSnapshot[applied={T|T < 7333120533837295616}]
I20260925 05:03:50.362250 12951 mvcc.cc:205] Tried to move back new op lower bound from 7333120533837295616 to 7333120533663551488. Current Snapshot: MvccSnapshot[applied={T|T < 7333120533837295616}]
I20260925 05:03:50.363219 12831 raft_consensus.cc:348] T 00000000000000000000000000000000 P db73e26a10314db2963941a9662e8b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45031 } } peers { permanent_uuid: "db73e26a10314db2963941a9662e8b74" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 } } peers { permanent_uuid: "ead8c59838dc4a04b8a4233c2470d516" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } }
I20260925 05:03:50.363624 12831 raft_consensus.cc:374] T 00000000000000000000000000000000 P db73e26a10314db2963941a9662e8b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:03:50.363755 12831 raft_consensus.cc:729] T 00000000000000000000000000000000 P db73e26a10314db2963941a9662e8b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db73e26a10314db2963941a9662e8b74, State: Initialized, Role: FOLLOWER
I20260925 05:03:50.364614 12831 consensus_queue.cc:260] T 00000000000000000000000000000000 P db73e26a10314db2963941a9662e8b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45031 } } peers { permanent_uuid: "db73e26a10314db2963941a9662e8b74" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 } } peers { permanent_uuid: "ead8c59838dc4a04b8a4233c2470d516" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } }
I20260925 05:03:50.365517 12933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45031 } } peers { permanent_uuid: "db73e26a10314db2963941a9662e8b74" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 } } peers { permanent_uuid: "ead8c59838dc4a04b8a4233c2470d516" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } } }
I20260925 05:03:50.365954 12933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [sys.catalog]: This master's current role is: LEADER
I20260925 05:03:50.367619 12948 catalog_manager.cc:1347] Generated new cluster ID: feb90de773fb48429705e20f9baed05e
I20260925 05:03:50.367717 12948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:03:50.382301 12919 sys_catalog.cc:455] T 00000000000000000000000000000000 P ead8c59838dc4a04b8a4233c2470d516 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45031 } } peers { permanent_uuid: "db73e26a10314db2963941a9662e8b74" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 } } peers { permanent_uuid: "ead8c59838dc4a04b8a4233c2470d516" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } } }
I20260925 05:03:50.382562 12919 sys_catalog.cc:458] T 00000000000000000000000000000000 P ead8c59838dc4a04b8a4233c2470d516 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:03:50.383803 12955 sys_catalog.cc:455] T 00000000000000000000000000000000 P db73e26a10314db2963941a9662e8b74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45031 } } peers { permanent_uuid: "db73e26a10314db2963941a9662e8b74" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 } } peers { permanent_uuid: "ead8c59838dc4a04b8a4233c2470d516" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } } }
I20260925 05:03:50.388178 12804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db73e26a10314db2963941a9662e8b74 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:03:50.388353 12806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db73e26a10314db2963941a9662e8b74" is_pre_election: true
I20260925 05:03:50.388672 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db73e26a10314db2963941a9662e8b74"
I20260925 05:03:50.397130 12804 raft_consensus.cc:1261] T 00000000000000000000000000000000 P db73e26a10314db2963941a9662e8b74 [term 1 FOLLOWER]: Refusing update from remote peer 7e6b9bf4a0744d6c9e9408e746d7f7b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:03:50.388864 12831 sys_catalog.cc:564] T 00000000000000000000000000000000 P db73e26a10314db2963941a9662e8b74 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:03:50.398327 12933 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db73e26a10314db2963941a9662e8b74" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:03:50.398619 12955 sys_catalog.cc:458] T 00000000000000000000000000000000 P db73e26a10314db2963941a9662e8b74 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:03:50.403820 12948 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:03:50.405561 12948 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:03:50.428237 12948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4: Generated new TSK 0
I20260925 05:03:50.429246 12948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:03:50.458168 12933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45031 } } peers { permanent_uuid: "db73e26a10314db2963941a9662e8b74" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 } } peers { permanent_uuid: "ead8c59838dc4a04b8a4233c2470d516" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } } }
I20260925 05:03:50.458411 12933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6b9bf4a0744d6c9e9408e746d7f7b4 [sys.catalog]: This master's current role is: LEADER
I20260925 05:03:50.458133 12961 sys_catalog.cc:455] T 00000000000000000000000000000000 P db73e26a10314db2963941a9662e8b74 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45031 } } peers { permanent_uuid: "db73e26a10314db2963941a9662e8b74" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 } } peers { permanent_uuid: "ead8c59838dc4a04b8a4233c2470d516" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } } }
I20260925 05:03:50.458526 12961 sys_catalog.cc:458] T 00000000000000000000000000000000 P db73e26a10314db2963941a9662e8b74 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:03:50.458858 12955 sys_catalog.cc:455] T 00000000000000000000000000000000 P db73e26a10314db2963941a9662e8b74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e6b9bf4a0744d6c9e9408e746d7f7b4. Latest consensus state: current_term: 1 leader_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6b9bf4a0744d6c9e9408e746d7f7b4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45031 } } peers { permanent_uuid: "db73e26a10314db2963941a9662e8b74" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 34843 } } peers { permanent_uuid: "ead8c59838dc4a04b8a4233c2470d516" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 40731 } } }
I20260925 05:03:50.459120 12955 sys_catalog.cc:458] T 00000000000000000000000000000000 P db73e26a10314db2963941a9662e8b74 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:03:50.541000 12975 catalog_manager.cc:1259] Loaded cluster ID: feb90de773fb48429705e20f9baed05e
I20260925 05:03:50.541157 12975 catalog_manager.cc:1552] T 00000000000000000000000000000000 P db73e26a10314db2963941a9662e8b74: loading cluster ID for follower catalog manager: success
I20260925 05:03:50.544692 12975 catalog_manager.cc:1574] T 00000000000000000000000000000000 P db73e26a10314db2963941a9662e8b74: acquiring CA information for follower catalog manager: success
I20260925 05:03:50.547032 12975 catalog_manager.cc:1602] T 00000000000000000000000000000000 P db73e26a10314db2963941a9662e8b74: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:03:51.149631 12930 catalog_manager.cc:1259] Loaded cluster ID: feb90de773fb48429705e20f9baed05e
I20260925 05:03:51.149772 12930 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ead8c59838dc4a04b8a4233c2470d516: loading cluster ID for follower catalog manager: success
I20260925 05:03:51.152191 12930 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ead8c59838dc4a04b8a4233c2470d516: acquiring CA information for follower catalog manager: success
I20260925 05:03:51.153890 12930 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ead8c59838dc4a04b8a4233c2470d516: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:03:51.590936 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 12690
I20260925 05:03:51.592175 12752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:03:51.705781 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 12690
I20260925 05:03:51.726454 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 12764
I20260925 05:03:51.727581 12826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:03:51.834352 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 12764
I20260925 05:03:51.854449 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 12838
I20260925 05:03:51.855532 12900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:03:51.961437 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 12838
2026-09-25T05:03:51Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4611 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:03:52 dist-test-slave-5n86 krb5kdc[12983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:03:52 dist-test-slave-5n86 krb5kdc[12983](info): set up 2 sockets
Sep 25 05:03:52 dist-test-slave-5n86 krb5kdc[12983](info): commencing operation
krb5kdc: starting...
W20260925 05:03:53.065274 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:03:53 dist-test-slave-5n86 krb5kdc[12983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:03:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:03:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:03:53.294394 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:34609
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:38843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:34609
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:03:53.615805 12996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:03:53.616169 12996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:03:53.616313 12996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:03:53.623551 12996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:03:53.623693 12996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:03:53.623795 12996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:03:53.623889 12996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:03:53.623960 12996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:03:53.633845 12996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:38843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:34609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:34609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:03:53.635222 12996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:03:53.636791 12996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:03:53.653106 13002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:53.653134 13001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:53.653174 13004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:03:53.654130 12996 server_base.cc:1034] running on GCE node
I20260925 05:03:53.654913 12996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:03:53.656337 12996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:03:53.657543 12996 hybrid_clock.cc:648] HybridClock initialized: now 1790312633657505 us; error 65 us; skew 500 ppm
I20260925 05:03:53.657984 12996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:03:53 dist-test-slave-5n86 krb5kdc[12983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312633, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:03:53.661206 12996 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:03:53.662729 12996 webserver.cc:460] Webserver started at http://127.30.247.62:45611/ using document root <none> and password file <none>
I20260925 05:03:53.663436 12996 fs_manager.cc:359] Metadata directory not provided
I20260925 05:03:53.663576 12996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:03:53.663882 12996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:03:53.666915 12996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "5cfd1f01ff874efaa1d1fc294cfb0e80"
format_stamp: "Formatted at 2026-09-25 05:03:53 on dist-test-slave-5n86"
server_key: "682e97638bba654e41394862d5ce698e"
server_key_iv: "b250371667a29c8611a0cad50508706c"
server_key_version: "encryptionkey@0"
I20260925 05:03:53.667661 12996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "5cfd1f01ff874efaa1d1fc294cfb0e80"
format_stamp: "Formatted at 2026-09-25 05:03:53 on dist-test-slave-5n86"
server_key: "682e97638bba654e41394862d5ce698e"
server_key_iv: "b250371667a29c8611a0cad50508706c"
server_key_version: "encryptionkey@0"
I20260925 05:03:53.672775 12996 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 05:03:53.676107 13011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:53.677471 12996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:03:53.677647 12996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "5cfd1f01ff874efaa1d1fc294cfb0e80"
format_stamp: "Formatted at 2026-09-25 05:03:53 on dist-test-slave-5n86"
server_key: "682e97638bba654e41394862d5ce698e"
server_key_iv: "b250371667a29c8611a0cad50508706c"
server_key_version: "encryptionkey@0"
I20260925 05:03:53.677836 12996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:03:53.720211 12996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:03:53.724435 12996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:03:53.724749 12996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:03:53.743472 12996 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:34609
I20260925 05:03:53.743501 13062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:34609 every 8 connection(s)
I20260925 05:03:53.744851 12996 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:03:53.745616 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 12996
I20260925 05:03:53.745887 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:03:53.746685 31708 external_mini_cluster.cc:1428] Setting key 4204bd49a1904f646b136248ffe443a4
I20260925 05:03:53.749811 13063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:03:53.764671 13063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cfd1f01ff874efaa1d1fc294cfb0e80: Bootstrap starting.
Sep 25 05:03:53 dist-test-slave-5n86 krb5kdc[12983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:03:53.768132 13063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cfd1f01ff874efaa1d1fc294cfb0e80: Neither blocks nor log segments found. Creating new log.
I20260925 05:03:53.769285 13063 log.cc:824] T 00000000000000000000000000000000 P 5cfd1f01ff874efaa1d1fc294cfb0e80: Log is configured to *not* fsync() on all Append() calls
I20260925 05:03:53.772113 13063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cfd1f01ff874efaa1d1fc294cfb0e80: No bootstrap required, opened a new log
I20260925 05:03:53.774385 13066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:53.748561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50952 (local address 127.30.247.62:34609)
0925 05:03:53.750025 (+ 1464us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:53.750057 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:53.750117 (+ 60us) server_negotiation.cc:407] Connection header received
0925 05:03:53.752524 (+ 2407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:53.752577 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:53.753061 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:53.755296 (+ 2235us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:53.755946 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:53.757902 (+ 1956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:53.759905 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:53.761056 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:53.761209 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:53.766337 (+ 5128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:53.766360 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:53.766385 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:53.766467 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:53.769691 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:53.770695 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:53.770702 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:53.770708 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:53.770856 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:53.771261 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:53.771266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:53.771268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:53.771762 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:53.772091 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:53.772605 (+ 514us) server_negotiation.cc:299] Negotiation successful
0925 05:03:53.773067 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1185,"thread_start_us":861,"threads_started":1}
I20260925 05:03:53.778384 31708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:03:53.780965 13063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cfd1f01ff874efaa1d1fc294cfb0e80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cfd1f01ff874efaa1d1fc294cfb0e80" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 34609 } }
I20260925 05:03:53.781313 13063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cfd1f01ff874efaa1d1fc294cfb0e80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:03:53.781414 13063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cfd1f01ff874efaa1d1fc294cfb0e80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cfd1f01ff874efaa1d1fc294cfb0e80, State: Initialized, Role: FOLLOWER
I20260925 05:03:53.781992 13063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cfd1f01ff874efaa1d1fc294cfb0e80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cfd1f01ff874efaa1d1fc294cfb0e80" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 34609 } }
I20260925 05:03:53.782217 13063 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cfd1f01ff874efaa1d1fc294cfb0e80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:03:53.782317 13063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cfd1f01ff874efaa1d1fc294cfb0e80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:03:53.782436 13063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cfd1f01ff874efaa1d1fc294cfb0e80 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:03:53.784489 13063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cfd1f01ff874efaa1d1fc294cfb0e80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cfd1f01ff874efaa1d1fc294cfb0e80" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 34609 } }
I20260925 05:03:53.784974 13063 leader_election.cc:302] T 00000000000000000000000000000000 P 5cfd1f01ff874efaa1d1fc294cfb0e80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cfd1f01ff874efaa1d1fc294cfb0e80; no voters:
I20260925 05:03:53.785488 13063 leader_election.cc:288] T 00000000000000000000000000000000 P 5cfd1f01ff874efaa1d1fc294cfb0e80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:03:53.785647 13068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cfd1f01ff874efaa1d1fc294cfb0e80 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:03:53.786345 13068 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cfd1f01ff874efaa1d1fc294cfb0e80 [term 1 LEADER]: Becoming Leader. State: Replica: 5cfd1f01ff874efaa1d1fc294cfb0e80, State: Running, Role: LEADER
I20260925 05:03:53.786898 13068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cfd1f01ff874efaa1d1fc294cfb0e80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cfd1f01ff874efaa1d1fc294cfb0e80" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 34609 } }
I20260925 05:03:53.787487 13063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cfd1f01ff874efaa1d1fc294cfb0e80 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:03:53.789901 13070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cfd1f01ff874efaa1d1fc294cfb0e80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cfd1f01ff874efaa1d1fc294cfb0e80. Latest consensus state: current_term: 1 leader_uuid: "5cfd1f01ff874efaa1d1fc294cfb0e80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cfd1f01ff874efaa1d1fc294cfb0e80" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 34609 } } }
I20260925 05:03:53.789872 13069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cfd1f01ff874efaa1d1fc294cfb0e80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cfd1f01ff874efaa1d1fc294cfb0e80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cfd1f01ff874efaa1d1fc294cfb0e80" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 34609 } } }
I20260925 05:03:53.790309 13070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cfd1f01ff874efaa1d1fc294cfb0e80 [sys.catalog]: This master's current role is: LEADER
I20260925 05:03:53.790318 13069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cfd1f01ff874efaa1d1fc294cfb0e80 [sys.catalog]: This master's current role is: LEADER
I20260925 05:03:53.792692 13073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:03:53.798286 13073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:03:53.806493 13073 catalog_manager.cc:1347] Generated new cluster ID: 6f5a08b2a7624f189d138efcdeafb1ee
I20260925 05:03:53.806624 13073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:03:53.821729 13073 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:03:53.823524 13073 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:03:53.846761 13073 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cfd1f01ff874efaa1d1fc294cfb0e80: Generated new TSK 0
I20260925 05:03:53.847831 13073 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:03:53.890648 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 12996
I20260925 05:03:53.891913 13058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:03:53.990854 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 12996
2026-09-25T05:03:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2024 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:03:54 dist-test-slave-5n86 krb5kdc[13091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:03:54 dist-test-slave-5n86 krb5kdc[13091](info): set up 2 sockets
Sep 25 05:03:54 dist-test-slave-5n86 krb5kdc[13091](info): commencing operation
krb5kdc: starting...
W20260925 05:03:55.081210 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:03:55 dist-test-slave-5n86 krb5kdc[13091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:03:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:03:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:03:55.314354 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:40301
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:40301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:03:55.636569 13104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:03:55.636894 13104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:03:55.637056 13104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:03:55.644124 13104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:03:55.644265 13104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:03:55.644368 13104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:03:55.644484 13104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:03:55.644587 13104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:03:55.654126 13104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:40301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:40301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:03:55.655450 13104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:03:55.656811 13104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:03:55.668447 13110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:55.668524 13112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:55.668488 13109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:03:55.669341 13104 server_base.cc:1034] running on GCE node
I20260925 05:03:55.670075 13104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:03:55.671434 13104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:03:55.672663 13104 hybrid_clock.cc:648] HybridClock initialized: now 1790312635672604 us; error 41 us; skew 500 ppm
I20260925 05:03:55.672989 13104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:03:55 dist-test-slave-5n86 krb5kdc[13091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312635, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:03:55.685129 13104 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:03:55.686661 13104 webserver.cc:460] Webserver started at http://127.30.247.62:35691/ using document root <none> and password file <none>
I20260925 05:03:55.687366 13104 fs_manager.cc:359] Metadata directory not provided
I20260925 05:03:55.687502 13104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:03:55.687808 13104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:03:55.690848 13104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "4efda22cdb06437995f8dd492a5bc3ee"
format_stamp: "Formatted at 2026-09-25 05:03:55 on dist-test-slave-5n86"
server_key: "f3e74dd0378b29d0053b0f90d26495d4"
server_key_iv: "fb0c72d25da2803c9dbc3efda0003eb6"
server_key_version: "encryptionkey@0"
I20260925 05:03:55.691598 13104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "4efda22cdb06437995f8dd492a5bc3ee"
format_stamp: "Formatted at 2026-09-25 05:03:55 on dist-test-slave-5n86"
server_key: "f3e74dd0378b29d0053b0f90d26495d4"
server_key_iv: "fb0c72d25da2803c9dbc3efda0003eb6"
server_key_version: "encryptionkey@0"
I20260925 05:03:55.697001 13104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 05:03:55.700338 13119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:55.701683 13104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:03:55.701856 13104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "4efda22cdb06437995f8dd492a5bc3ee"
format_stamp: "Formatted at 2026-09-25 05:03:55 on dist-test-slave-5n86"
server_key: "f3e74dd0378b29d0053b0f90d26495d4"
server_key_iv: "fb0c72d25da2803c9dbc3efda0003eb6"
server_key_version: "encryptionkey@0"
I20260925 05:03:55.702045 13104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:03:55.720052 13104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:03:55.724501 13104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:03:55.724838 13104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:03:55.743146 13104 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:40301
I20260925 05:03:55.743204 13170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:40301 every 8 connection(s)
I20260925 05:03:55.744508 13104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:03:55.746021 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 13104
I20260925 05:03:55.746241 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:03:55.747112 31708 external_mini_cluster.cc:1428] Setting key d9cd67fa1da103fa2f1125baf84ebffe
I20260925 05:03:55.749181 13171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:03:55.760108 13171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee: Bootstrap starting.
Sep 25 05:03:55 dist-test-slave-5n86 krb5kdc[13091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:03:55.763258 13171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee: Neither blocks nor log segments found. Creating new log.
I20260925 05:03:55.764293 13171 log.cc:824] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee: Log is configured to *not* fsync() on all Append() calls
I20260925 05:03:55.767928 13171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee: No bootstrap required, opened a new log
I20260925 05:03:55.769824 13174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:55.749272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44152 (local address 127.30.247.62:40301)
0925 05:03:55.750013 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:55.750025 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:55.750065 (+ 40us) server_negotiation.cc:407] Connection header received
0925 05:03:55.751666 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:55.751687 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:55.752080 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:55.752469 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:55.753939 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:55.755276 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:55.756586 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:55.757290 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:55.757496 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:55.761583 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:55.761591 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:55.761605 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:55.761642 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:55.764911 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:55.765793 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:55.765801 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:55.765809 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:55.766028 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:55.766554 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:55.766561 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:55.766565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:55.767369 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:55.767677 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:55.768189 (+ 512us) server_negotiation.cc:299] Negotiation successful
0925 05:03:55.768660 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":327,"threads_started":1}
I20260925 05:03:55.776686 13171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efda22cdb06437995f8dd492a5bc3ee" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40301 } }
I20260925 05:03:55.777077 13171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:03:55.777175 13171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4efda22cdb06437995f8dd492a5bc3ee, State: Initialized, Role: FOLLOWER
I20260925 05:03:55.777776 13171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efda22cdb06437995f8dd492a5bc3ee" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40301 } }
I20260925 05:03:55.778013 13171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:03:55.778146 13171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:03:55.778309 13171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:03:55.781067 13171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efda22cdb06437995f8dd492a5bc3ee" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40301 } }
I20260925 05:03:55.781553 13171 leader_election.cc:302] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4efda22cdb06437995f8dd492a5bc3ee; no voters:
I20260925 05:03:55.782130 13171 leader_election.cc:288] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:03:55.782281 13176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:03:55.782953 13176 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [term 1 LEADER]: Becoming Leader. State: Replica: 4efda22cdb06437995f8dd492a5bc3ee, State: Running, Role: LEADER
I20260925 05:03:55.783442 13176 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efda22cdb06437995f8dd492a5bc3ee" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40301 } }
I20260925 05:03:55.784132 13171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:03:55.786268 13178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4efda22cdb06437995f8dd492a5bc3ee. Latest consensus state: current_term: 1 leader_uuid: "4efda22cdb06437995f8dd492a5bc3ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efda22cdb06437995f8dd492a5bc3ee" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40301 } } }
I20260925 05:03:55.786708 13178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [sys.catalog]: This master's current role is: LEADER
I20260925 05:03:55.786989 13177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4efda22cdb06437995f8dd492a5bc3ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efda22cdb06437995f8dd492a5bc3ee" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40301 } } }
I20260925 05:03:55.787333 13177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [sys.catalog]: This master's current role is: LEADER
I20260925 05:03:55.788004 13185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:03:55.793614 13185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:03:55.802614 13185 catalog_manager.cc:1347] Generated new cluster ID: 4a7bb354670f46128f2bb0f015907101
I20260925 05:03:55.802729 13185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:03:55.826282 13185 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:03:55.827613 13185 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:03:55.845885 13185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee: Generated new TSK 0
I20260925 05:03:55.846762 13185 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
I20260925 05:03:55.910004 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:0
--local_ip_for_outbound_sockets=127.30.247.1
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:40301
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:03:56.269608 13197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:03:56.269871 13197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:03:56.269963 13197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:03:56.276925 13197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:03:56.277022 13197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:03:56.277132 13197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:03:56.286659 13197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:40301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:03:56.287879 13197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:03:56.289323 13197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:03:56.301819 13203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:56.301862 13205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:56.301867 13202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:03:56.303009 13197 server_base.cc:1034] running on GCE node
I20260925 05:03:56.303560 13197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:03:56.304418 13197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:03:56.305641 13197 hybrid_clock.cc:648] HybridClock initialized: now 1790312636305589 us; error 66 us; skew 500 ppm
I20260925 05:03:56.305931 13197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:03:56 dist-test-slave-5n86 krb5kdc[13091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312636, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:03:56.308941 13197 init.cc:377] Logged in from keytab as kudu/127.30.247.1@KRBTEST.COM (short username kudu)
I20260925 05:03:56.310220 13197 webserver.cc:460] Webserver started at http://127.30.247.1:38093/ using document root <none> and password file <none>
I20260925 05:03:56.310815 13197 fs_manager.cc:359] Metadata directory not provided
I20260925 05:03:56.310909 13197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:03:56.311163 13197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:03:56.313854 13197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/data/instance:
uuid: "5354129b30144953beced6758ed463e8"
format_stamp: "Formatted at 2026-09-25 05:03:56 on dist-test-slave-5n86"
server_key: "139d48cb069de017c59e2cdf9c5ee6d0"
server_key_iv: "fc4024edeed59fbc74f761b637b6d65e"
server_key_version: "encryptionkey@0"
I20260925 05:03:56.314564 13197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance:
uuid: "5354129b30144953beced6758ed463e8"
format_stamp: "Formatted at 2026-09-25 05:03:56 on dist-test-slave-5n86"
server_key: "139d48cb069de017c59e2cdf9c5ee6d0"
server_key_iv: "fc4024edeed59fbc74f761b637b6d65e"
server_key_version: "encryptionkey@0"
I20260925 05:03:56.319444 13197 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 05:03:56.322736 13212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:56.324127 13197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 05:03:56.324302 13197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "5354129b30144953beced6758ed463e8"
format_stamp: "Formatted at 2026-09-25 05:03:56 on dist-test-slave-5n86"
server_key: "139d48cb069de017c59e2cdf9c5ee6d0"
server_key_iv: "fc4024edeed59fbc74f761b637b6d65e"
server_key_version: "encryptionkey@0"
I20260925 05:03:56.324493 13197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:03:56.353070 13197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:03:56.356922 13197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:03:56.357216 13197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:03:56.358110 13197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:03:56.359364 13197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:03:56.359439 13197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:56.359601 13197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:03:56.359679 13197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:56.391892 13197 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:45219
I20260925 05:03:56.391978 13324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:45219 every 8 connection(s)
I20260925 05:03:56.393213 13197 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:03:56.396265 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 13197
I20260925 05:03:56.396477 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance
I20260925 05:03:56.397135 31708 external_mini_cluster.cc:1428] Setting key 39b762e12cb7ca3defb406f5b674ccfa
Sep 25 05:03:56 dist-test-slave-5n86 krb5kdc[13091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312636, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:03:56.413750 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:56.395696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:57625 (local address 127.30.247.62:40301)
0925 05:03:56.396203 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:56.396210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:56.397298 (+ 1088us) server_negotiation.cc:407] Connection header received
0925 05:03:56.399275 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:56.399280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:56.399360 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:56.399501 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:56.401053 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:56.402229 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:56.403808 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:56.404525 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:56.404697 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:56.408229 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:56.408234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:56.408237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:56.408267 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:56.410810 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:56.411505 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:56.411512 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:56.411516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:56.411630 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:56.412036 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:56.412042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:56.412045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:56.412290 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:56.412444 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:56.413163 (+ 719us) server_negotiation.cc:299] Negotiation successful
0925 05:03:56.413383 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":265,"threads_started":1}
I20260925 05:03:56.414659 13328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:56.396290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.1:57625)
0925 05:03:56.397148 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:56.397193 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:56.399026 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:56.399756 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:56.399764 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:56.400252 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:03:56.400888 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:56.400907 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:56.402404 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:56.402408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:56.403658 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:56.403671 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:56.404756 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:56.404761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:56.405008 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:56.405707 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:03:56.405740 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:03:56.408032 (+ 2292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:03:56.410963 (+ 2931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:56.410971 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:56.410987 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:56.411378 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:56.411731 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:56.411737 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:56.411740 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:56.411923 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:56.412415 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:03:56.412423 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:03:56.412802 (+ 379us) client_negotiation.cc:769] Sending connection context
0925 05:03:56.413059 (+ 257us) client_negotiation.cc:240] Negotiation successful
0925 05:03:56.413556 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":379,"threads_started":1}
I20260925 05:03:56.416322 13325 heartbeater.cc:347] Connected to a master server at 127.30.247.62:40301
I20260925 05:03:56.416663 13325 heartbeater.cc:464] Registering TS with master...
I20260925 05:03:56.417372 13325 heartbeater.cc:511] Master 127.30.247.62:40301 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.30.247.2@KRBTEST.COM; defaulting to no policy
I20260925 05:03:56.419438 13136 ts_manager.cc:194] Registered new tserver with Master: 5354129b30144953beced6758ed463e8 (127.30.247.1:45219)
I20260925 05:03:56.421084 13136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.1@KRBTEST.COM'} at 127.30.247.1:57625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
I20260925 05:03:56.448482 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:0
--local_ip_for_outbound_sockets=127.30.247.2
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:40301
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:03:56.766899 13331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:03:56.767220 13331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:03:56.767318 13331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:03:56.776178 13331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:03:56.776286 13331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:03:56.776399 13331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:03:56.786787 13331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:40301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:03:56.788358 13331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:03:56.789820 13331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:03:56.803086 13339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:56.803109 13337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:56.803212 13336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:03:56.803612 13331 server_base.cc:1034] running on GCE node
I20260925 05:03:56.804411 13331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:03:56.805483 13331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:03:56.806711 13331 hybrid_clock.cc:648] HybridClock initialized: now 1790312636806649 us; error 51 us; skew 500 ppm
I20260925 05:03:56.807035 13331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:03:56 dist-test-slave-5n86 krb5kdc[13091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312636, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:03:56.810129 13331 init.cc:377] Logged in from keytab as kudu/127.30.247.2@KRBTEST.COM (short username kudu)
I20260925 05:03:56.811561 13331 webserver.cc:460] Webserver started at http://127.30.247.2:34681/ using document root <none> and password file <none>
I20260925 05:03:56.812250 13331 fs_manager.cc:359] Metadata directory not provided
I20260925 05:03:56.812392 13331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:03:56.812744 13331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:03:56.815687 13331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/data/instance:
uuid: "11fb070ae4d147dd85574dbcc1310515"
format_stamp: "Formatted at 2026-09-25 05:03:56 on dist-test-slave-5n86"
server_key: "c1fa50450e682937edb3326a05054bc6"
server_key_iv: "69c2d262085337c5f00d30270626e22a"
server_key_version: "encryptionkey@0"
I20260925 05:03:56.816500 13331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance:
uuid: "11fb070ae4d147dd85574dbcc1310515"
format_stamp: "Formatted at 2026-09-25 05:03:56 on dist-test-slave-5n86"
server_key: "c1fa50450e682937edb3326a05054bc6"
server_key_iv: "69c2d262085337c5f00d30270626e22a"
server_key_version: "encryptionkey@0"
I20260925 05:03:56.821838 13331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 05:03:56.825403 13346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:56.826772 13331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:03:56.826960 13331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "11fb070ae4d147dd85574dbcc1310515"
format_stamp: "Formatted at 2026-09-25 05:03:56 on dist-test-slave-5n86"
server_key: "c1fa50450e682937edb3326a05054bc6"
server_key_iv: "69c2d262085337c5f00d30270626e22a"
server_key_version: "encryptionkey@0"
I20260925 05:03:56.827178 13331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:03:56.841775 13331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:03:56.845837 13331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:03:56.846129 13331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:03:56.847046 13331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:03:56.848337 13331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:03:56.848439 13331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:56.848606 13331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:03:56.848681 13331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:56.881414 13331 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:35113
I20260925 05:03:56.881470 13458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:35113 every 8 connection(s)
I20260925 05:03:56.882673 13331 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:03:56.889210 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 13331
I20260925 05:03:56.889412 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance
I20260925 05:03:56.889916 31708 external_mini_cluster.cc:1428] Setting key ebd07a6f2442031dc79918402f2f61ec
Sep 25 05:03:56 dist-test-slave-5n86 krb5kdc[13091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312636, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:03:56.904371 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:56.885187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:60695 (local address 127.30.247.62:40301)
0925 05:03:56.885352 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:56.885356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:56.886600 (+ 1244us) server_negotiation.cc:407] Connection header received
0925 05:03:56.888469 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:56.888476 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:56.888604 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:56.888709 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:56.890272 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:56.891325 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:56.893680 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:56.894088 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:56.894183 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:56.898353 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:56.898358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:56.898361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:56.898395 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:56.901477 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:56.902154 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:56.902160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:56.902163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:56.902289 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:56.902703 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:56.902708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:56.902712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:56.902980 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:56.903126 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:56.903785 (+ 659us) server_negotiation.cc:299] Negotiation successful
0925 05:03:56.904019 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":53}
I20260925 05:03:56.905321 13461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:56.885743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.2:60695)
0925 05:03:56.886447 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:56.886484 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:56.888159 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:56.888955 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:56.888964 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:56.889518 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:03:56.890117 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:56.890135 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:56.891457 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:56.891461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:56.893513 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:56.893532 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:56.894301 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:56.894310 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:56.894642 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:56.895360 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:03:56.895395 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:03:56.898137 (+ 2742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:03:56.901611 (+ 3474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:56.901620 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:56.901635 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:56.902029 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:56.902404 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:56.902409 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:56.902413 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:56.902595 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:56.903110 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:03:56.903118 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:03:56.903429 (+ 311us) client_negotiation.cc:769] Sending connection context
0925 05:03:56.903682 (+ 253us) client_negotiation.cc:240] Negotiation successful
0925 05:03:56.904192 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":274,"threads_started":1}
I20260925 05:03:56.906869 13459 heartbeater.cc:347] Connected to a master server at 127.30.247.62:40301
I20260925 05:03:56.907202 13459 heartbeater.cc:464] Registering TS with master...
I20260925 05:03:56.907905 13459 heartbeater.cc:511] Master 127.30.247.62:40301 requested a full tablet report, sending...
I20260925 05:03:56.909368 13136 ts_manager.cc:194] Registered new tserver with Master: 11fb070ae4d147dd85574dbcc1310515 (127.30.247.2:35113)
I20260925 05:03:56.910615 13136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.2@KRBTEST.COM'} at 127.30.247.2:60695
WARNING: no policy specified for kudu/127.30.247.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
I20260925 05:03:56.941612 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:0
--local_ip_for_outbound_sockets=127.30.247.3
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:40301
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:03:57.261534 13464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:03:57.261807 13464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:03:57.261978 13464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:03:57.269099 13464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:03:57.269193 13464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:03:57.269392 13464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:03:57.279076 13464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:40301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:03:57.280267 13464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:03:57.281616 13464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:03:57.294540 13469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:57.294591 13470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:57.294608 13472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:03:57.295331 13464 server_base.cc:1034] running on GCE node
I20260925 05:03:57.295879 13464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:03:57.296749 13464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:03:57.297935 13464 hybrid_clock.cc:648] HybridClock initialized: now 1790312637297878 us; error 54 us; skew 500 ppm
I20260925 05:03:57.298231 13464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:03:57 dist-test-slave-5n86 krb5kdc[13091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312637, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:03:57.301240 13464 init.cc:377] Logged in from keytab as kudu/127.30.247.3@KRBTEST.COM (short username kudu)
I20260925 05:03:57.302632 13464 webserver.cc:460] Webserver started at http://127.30.247.3:34851/ using document root <none> and password file <none>
I20260925 05:03:57.303247 13464 fs_manager.cc:359] Metadata directory not provided
I20260925 05:03:57.303344 13464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:03:57.303580 13464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:03:57.306386 13464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/data/instance:
uuid: "3c8b7b49316b439896cac8f385b46bc5"
format_stamp: "Formatted at 2026-09-25 05:03:57 on dist-test-slave-5n86"
server_key: "3f01e680bd34b2de6b6639cb80a695ad"
server_key_iv: "c8a8adde32ffa48dc58f969eacee0ce0"
server_key_version: "encryptionkey@0"
I20260925 05:03:57.307158 13464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance:
uuid: "3c8b7b49316b439896cac8f385b46bc5"
format_stamp: "Formatted at 2026-09-25 05:03:57 on dist-test-slave-5n86"
server_key: "3f01e680bd34b2de6b6639cb80a695ad"
server_key_iv: "c8a8adde32ffa48dc58f969eacee0ce0"
server_key_version: "encryptionkey@0"
I20260925 05:03:57.312083 13464 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 05:03:57.315405 13479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:57.316747 13464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:03:57.316905 13464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "3c8b7b49316b439896cac8f385b46bc5"
format_stamp: "Formatted at 2026-09-25 05:03:57 on dist-test-slave-5n86"
server_key: "3f01e680bd34b2de6b6639cb80a695ad"
server_key_iv: "c8a8adde32ffa48dc58f969eacee0ce0"
server_key_version: "encryptionkey@0"
I20260925 05:03:57.317070 13464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:03:57.334702 13464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:03:57.338495 13464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:03:57.338784 13464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:03:57.339679 13464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:03:57.341147 13464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:03:57.341218 13464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:57.341364 13464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:03:57.341415 13464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:57.373858 13464 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:41937
I20260925 05:03:57.373941 13591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:41937 every 8 connection(s)
I20260925 05:03:57.375120 13464 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:03:57.381474 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 13464
I20260925 05:03:57.381673 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance
I20260925 05:03:57.382226 31708 external_mini_cluster.cc:1428] Setting key 152bccaa971e98f4414c13e1aa8cbf87
Sep 25 05:03:57 dist-test-slave-5n86 krb5kdc[13091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312637, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:03:57.395709 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:57.377581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:35889 (local address 127.30.247.62:40301)
0925 05:03:57.377743 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:57.377747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:57.378843 (+ 1096us) server_negotiation.cc:407] Connection header received
0925 05:03:57.380682 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:57.380688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:57.380777 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:57.380883 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:57.382865 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.383994 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.385882 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.386296 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.386418 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:57.389707 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:57.389712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:57.389717 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:57.389757 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:57.392741 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:57.393431 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:57.393437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:57.393440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:57.393609 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:57.394017 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:57.394022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:57.394025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:57.394364 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:57.394512 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:57.395147 (+ 635us) server_negotiation.cc:299] Negotiation successful
0925 05:03:57.395355 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 05:03:57.396689 13594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:57.378008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.3:35889)
0925 05:03:57.378682 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:57.378725 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:57.380355 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:57.381143 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:57.381157 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:57.381903 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:03:57.382684 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:57.382709 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.384149 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.384155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:57.385746 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:57.385757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.386515 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.386522 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:57.386757 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:57.387396 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:03:57.387423 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:03:57.389516 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:03:57.392875 (+ 3359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:57.392881 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:57.392891 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:57.393298 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:57.393722 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:57.393726 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:57.393728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:57.393894 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:57.394484 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:03:57.394489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:03:57.394760 (+ 271us) client_negotiation.cc:769] Sending connection context
0925 05:03:57.394977 (+ 217us) client_negotiation.cc:240] Negotiation successful
0925 05:03:57.395377 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":263,"threads_started":1}
I20260925 05:03:57.398126 13592 heartbeater.cc:347] Connected to a master server at 127.30.247.62:40301
I20260925 05:03:57.398406 13592 heartbeater.cc:464] Registering TS with master...
I20260925 05:03:57.399089 13592 heartbeater.cc:511] Master 127.30.247.62:40301 requested a full tablet report, sending...
I20260925 05:03:57.400494 13135 ts_manager.cc:194] Registered new tserver with Master: 3c8b7b49316b439896cac8f385b46bc5 (127.30.247.3:41937)
I20260925 05:03:57.401412 13135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.3@KRBTEST.COM'} at 127.30.247.3:35889
I20260925 05:03:57.409689 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:03:57.423867 13325 heartbeater.cc:503] Master 127.30.247.62:40301 was elected leader, sending a full tablet report...
I20260925 05:03:57.426183 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:57.414104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44158 (local address 127.30.247.62:40301)
0925 05:03:57.414253 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:57.414257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:57.414584 (+ 327us) server_negotiation.cc:407] Connection header received
0925 05:03:57.414750 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:57.414754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:57.414813 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:57.414911 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:57.416193 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.416978 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.418256 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.418680 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.418772 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:57.421011 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:57.421015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:57.421018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:57.421078 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:57.423474 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:57.424129 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:57.424135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:57.424140 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:57.424278 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:57.424743 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:57.424749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:57.424753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:57.425063 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:57.425332 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:57.425644 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 05:03:57.425887 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 05:03:57.430032 13135 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:03:57.437095 13135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:03:57.469376 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:57.448768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:41937 (local address 127.0.0.1:40270)
0925 05:03:57.459574 (+ 10806us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:57.459621 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:57.459805 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:57.460711 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:57.460719 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:57.460764 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:03:57.461550 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:57.461564 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.463611 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.463617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:57.466497 (+ 2880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:57.466519 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.468297 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.468303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:57.468521 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:57.468606 (+ 85us) client_negotiation.cc:769] Sending connection context
0925 05:03:57.468729 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 05:03:57.468912 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10661,"mutex_wait_us":235,"spinlock_wait_cycles":5504,"thread_start_us":10527,"threads_started":1}
I20260925 05:03:57.469683 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:57.448657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59122 (local address 127.30.247.2:35113)
0925 05:03:57.459083 (+ 10426us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:57.459094 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:57.459799 (+ 705us) server_negotiation.cc:407] Connection header received
0925 05:03:57.460046 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:57.460052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:57.460226 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:57.460408 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:03:57.461656 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.463394 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.466955 (+ 3561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.468349 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.468476 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:57.468701 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:57.469029 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 05:03:57.469192 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10235,"thread_start_us":10070,"threads_started":1}
I20260925 05:03:57.469561 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:57.448433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:35113 (local address 127.0.0.1:59122)
0925 05:03:57.459572 (+ 11139us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:57.459619 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:57.459804 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:57.460711 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:57.460719 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:57.460766 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:03:57.461429 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:57.461444 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.463593 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.463599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:57.466786 (+ 3187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:57.466798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.468663 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.468669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:57.468838 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:57.468861 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 05:03:57.468957 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 05:03:57.469090 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10885,"mutex_wait_us":254,"spinlock_wait_cycles":16128,"thread_start_us":10689,"threads_started":1}
I20260925 05:03:57.470101 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:57.449591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40270 (local address 127.30.247.3:41937)
0925 05:03:57.458734 (+ 9143us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:57.458746 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:57.459870 (+ 1124us) server_negotiation.cc:407] Connection header received
0925 05:03:57.459984 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:57.459992 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:57.460227 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:57.460412 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:03:57.461759 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.463456 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.466692 (+ 3236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.468111 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.468243 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:57.468475 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:57.469619 (+ 1144us) server_negotiation.cc:299] Negotiation successful
0925 05:03:57.469766 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8906,"thread_start_us":8720,"threads_started":1}
I20260925 05:03:57.474313 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:57.460111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:45219 (local address 127.0.0.1:47482)
0925 05:03:57.466607 (+ 6496us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:57.466625 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:57.466731 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:57.467780 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:57.467785 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:57.467809 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:03:57.468152 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:57.468160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.470051 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.470055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:57.472194 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:57.472207 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.473558 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.473564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:57.473728 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:57.473760 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 05:03:57.473873 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 05:03:57.473990 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6373,"thread_start_us":6718,"threads_started":1}
I20260925 05:03:57.474395 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:57.460238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47482 (local address 127.30.247.1:45219)
0925 05:03:57.467339 (+ 7101us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:57.467345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:57.467367 (+ 22us) server_negotiation.cc:407] Connection header received
0925 05:03:57.467436 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:57.467441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:57.467616 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:57.467796 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:03:57.468365 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.469920 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.472373 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.473358 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.473528 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:57.473742 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:57.473914 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 05:03:57.474064 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6960,"thread_start_us":6771,"threads_started":1}
I20260925 05:03:57.473865 13527 tablet_service.cc:1467] Processing CreateTablet for tablet bcc7d366dd484f779f3b644cfe3101eb (DEFAULT_TABLE table=test-table [id=b49a9874bda641f7a1df5d5085adc191]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:03:57.473865 13394 tablet_service.cc:1467] Processing CreateTablet for tablet bcc7d366dd484f779f3b644cfe3101eb (DEFAULT_TABLE table=test-table [id=b49a9874bda641f7a1df5d5085adc191]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:03:57.475265 13394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcc7d366dd484f779f3b644cfe3101eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:03:57.475261 13527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcc7d366dd484f779f3b644cfe3101eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:03:57.479418 13260 tablet_service.cc:1467] Processing CreateTablet for tablet bcc7d366dd484f779f3b644cfe3101eb (DEFAULT_TABLE table=test-table [id=b49a9874bda641f7a1df5d5085adc191]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:03:57.480649 13260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcc7d366dd484f779f3b644cfe3101eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:03:57.489898 13611 tablet_bootstrap.cc:492] T bcc7d366dd484f779f3b644cfe3101eb P 3c8b7b49316b439896cac8f385b46bc5: Bootstrap starting.
I20260925 05:03:57.490948 13610 tablet_bootstrap.cc:492] T bcc7d366dd484f779f3b644cfe3101eb P 11fb070ae4d147dd85574dbcc1310515: Bootstrap starting.
I20260925 05:03:57.492776 13612 tablet_bootstrap.cc:492] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8: Bootstrap starting.
I20260925 05:03:57.493885 13611 tablet_bootstrap.cc:654] T bcc7d366dd484f779f3b644cfe3101eb P 3c8b7b49316b439896cac8f385b46bc5: Neither blocks nor log segments found. Creating new log.
I20260925 05:03:57.494299 13610 tablet_bootstrap.cc:654] T bcc7d366dd484f779f3b644cfe3101eb P 11fb070ae4d147dd85574dbcc1310515: Neither blocks nor log segments found. Creating new log.
I20260925 05:03:57.494905 13611 log.cc:824] T bcc7d366dd484f779f3b644cfe3101eb P 3c8b7b49316b439896cac8f385b46bc5: Log is configured to *not* fsync() on all Append() calls
I20260925 05:03:57.495463 13610 log.cc:824] T bcc7d366dd484f779f3b644cfe3101eb P 11fb070ae4d147dd85574dbcc1310515: Log is configured to *not* fsync() on all Append() calls
I20260925 05:03:57.496043 13612 tablet_bootstrap.cc:654] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8: Neither blocks nor log segments found. Creating new log.
I20260925 05:03:57.497184 13612 log.cc:824] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8: Log is configured to *not* fsync() on all Append() calls
I20260925 05:03:57.497797 13611 tablet_bootstrap.cc:492] T bcc7d366dd484f779f3b644cfe3101eb P 3c8b7b49316b439896cac8f385b46bc5: No bootstrap required, opened a new log
I20260925 05:03:57.498101 13611 ts_tablet_manager.cc:1397] T bcc7d366dd484f779f3b644cfe3101eb P 3c8b7b49316b439896cac8f385b46bc5: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 05:03:57.499177 13610 tablet_bootstrap.cc:492] T bcc7d366dd484f779f3b644cfe3101eb P 11fb070ae4d147dd85574dbcc1310515: No bootstrap required, opened a new log
I20260925 05:03:57.499461 13610 ts_tablet_manager.cc:1397] T bcc7d366dd484f779f3b644cfe3101eb P 11fb070ae4d147dd85574dbcc1310515: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260925 05:03:57.503669 13612 tablet_bootstrap.cc:492] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8: No bootstrap required, opened a new log
I20260925 05:03:57.503914 13612 ts_tablet_manager.cc:1397] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260925 05:03:57.505657 13611 raft_consensus.cc:348] T bcc7d366dd484f779f3b644cfe3101eb P 3c8b7b49316b439896cac8f385b46bc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fb070ae4d147dd85574dbcc1310515" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35113 } } peers { permanent_uuid: "3c8b7b49316b439896cac8f385b46bc5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41937 } } peers { permanent_uuid: "5354129b30144953beced6758ed463e8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45219 } }
I20260925 05:03:57.506011 13611 raft_consensus.cc:374] T bcc7d366dd484f779f3b644cfe3101eb P 3c8b7b49316b439896cac8f385b46bc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:03:57.506129 13611 raft_consensus.cc:729] T bcc7d366dd484f779f3b644cfe3101eb P 3c8b7b49316b439896cac8f385b46bc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c8b7b49316b439896cac8f385b46bc5, State: Initialized, Role: FOLLOWER
I20260925 05:03:57.506757 13611 consensus_queue.cc:260] T bcc7d366dd484f779f3b644cfe3101eb P 3c8b7b49316b439896cac8f385b46bc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fb070ae4d147dd85574dbcc1310515" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35113 } } peers { permanent_uuid: "3c8b7b49316b439896cac8f385b46bc5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41937 } } peers { permanent_uuid: "5354129b30144953beced6758ed463e8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45219 } }
I20260925 05:03:57.507746 13592 heartbeater.cc:503] Master 127.30.247.62:40301 was elected leader, sending a full tablet report...
I20260925 05:03:57.509531 13611 ts_tablet_manager.cc:1428] T bcc7d366dd484f779f3b644cfe3101eb P 3c8b7b49316b439896cac8f385b46bc5: Time spent starting tablet: real 0.011s user 0.005s sys 0.004s
I20260925 05:03:57.510325 13610 raft_consensus.cc:348] T bcc7d366dd484f779f3b644cfe3101eb P 11fb070ae4d147dd85574dbcc1310515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fb070ae4d147dd85574dbcc1310515" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35113 } } peers { permanent_uuid: "3c8b7b49316b439896cac8f385b46bc5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41937 } } peers { permanent_uuid: "5354129b30144953beced6758ed463e8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45219 } }
I20260925 05:03:57.510743 13610 raft_consensus.cc:374] T bcc7d366dd484f779f3b644cfe3101eb P 11fb070ae4d147dd85574dbcc1310515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:03:57.510839 13610 raft_consensus.cc:729] T bcc7d366dd484f779f3b644cfe3101eb P 11fb070ae4d147dd85574dbcc1310515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11fb070ae4d147dd85574dbcc1310515, State: Initialized, Role: FOLLOWER
I20260925 05:03:57.511498 13610 consensus_queue.cc:260] T bcc7d366dd484f779f3b644cfe3101eb P 11fb070ae4d147dd85574dbcc1310515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fb070ae4d147dd85574dbcc1310515" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35113 } } peers { permanent_uuid: "3c8b7b49316b439896cac8f385b46bc5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41937 } } peers { permanent_uuid: "5354129b30144953beced6758ed463e8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45219 } }
I20260925 05:03:57.513235 13459 heartbeater.cc:503] Master 127.30.247.62:40301 was elected leader, sending a full tablet report...
I20260925 05:03:57.513228 13610 ts_tablet_manager.cc:1428] T bcc7d366dd484f779f3b644cfe3101eb P 11fb070ae4d147dd85574dbcc1310515: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260925 05:03:57.514153 13612 raft_consensus.cc:348] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fb070ae4d147dd85574dbcc1310515" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35113 } } peers { permanent_uuid: "3c8b7b49316b439896cac8f385b46bc5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41937 } } peers { permanent_uuid: "5354129b30144953beced6758ed463e8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45219 } }
I20260925 05:03:57.514564 13612 raft_consensus.cc:374] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:03:57.514688 13612 raft_consensus.cc:729] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5354129b30144953beced6758ed463e8, State: Initialized, Role: FOLLOWER
I20260925 05:03:57.515196 13612 consensus_queue.cc:260] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fb070ae4d147dd85574dbcc1310515" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35113 } } peers { permanent_uuid: "3c8b7b49316b439896cac8f385b46bc5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41937 } } peers { permanent_uuid: "5354129b30144953beced6758ed463e8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45219 } }
I20260925 05:03:57.516723 13612 ts_tablet_manager.cc:1428] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8: Time spent starting tablet: real 0.013s user 0.006s sys 0.008s
W20260925 05:03:57.626660 13593 tablet.cc:2369] T bcc7d366dd484f779f3b644cfe3101eb P 3c8b7b49316b439896cac8f385b46bc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:03:57.634568 13460 tablet.cc:2369] T bcc7d366dd484f779f3b644cfe3101eb P 11fb070ae4d147dd85574dbcc1310515: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:03:57.645493 13326 tablet.cc:2369] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:03:57.836858 13618 raft_consensus.cc:482] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:03:57.837169 13618 raft_consensus.cc:504] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fb070ae4d147dd85574dbcc1310515" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35113 } } peers { permanent_uuid: "3c8b7b49316b439896cac8f385b46bc5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41937 } } peers { permanent_uuid: "5354129b30144953beced6758ed463e8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45219 } }
I20260925 05:03:57.838852 13618 leader_election.cc:288] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11fb070ae4d147dd85574dbcc1310515 (127.30.247.2:35113), 3c8b7b49316b439896cac8f385b46bc5 (127.30.247.3:41937)
I20260925 05:03:57.845767 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:57.839379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:47071 (local address 127.30.247.2:35113)
0925 05:03:57.839630 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:57.839639 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:57.840458 (+ 819us) server_negotiation.cc:407] Connection header received
0925 05:03:57.840679 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:57.840683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:57.840745 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:57.840836 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:03:57.841455 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.842142 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.843900 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.844752 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.844946 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:57.845026 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:57.845341 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 05:03:57.845483 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 05:03:57.845779 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:57.839432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:50897 (local address 127.30.247.3:41937)
0925 05:03:57.839632 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:57.839639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:57.840484 (+ 845us) server_negotiation.cc:407] Connection header received
0925 05:03:57.840719 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:57.840724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:57.840791 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:57.840886 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:03:57.841485 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.842181 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.843924 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.844757 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.844953 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:57.845035 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:57.845383 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 05:03:57.845522 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 05:03:57.845925 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:57.839329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:35113 (local address 127.30.247.1:47071)
0925 05:03:57.840317 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:57.840332 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:57.840506 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:57.840856 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:57.840860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:57.840883 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:03:57.841280 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:57.841292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.842274 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.842281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:57.843748 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:57.843759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.844984 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.844990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:57.845176 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:57.845212 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 05:03:57.845315 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 05:03:57.845474 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"mutex_wait_us":29,"thread_start_us":534,"threads_started":1}
I20260925 05:03:57.846207 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:57.839364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:41937 (local address 127.30.247.1:50897)
0925 05:03:57.840380 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:57.840394 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:57.840541 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:57.840902 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:57.840906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:57.840928 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:03:57.841321 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:57.841329 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.842331 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.842336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:57.843777 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:57.843789 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.845023 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.845029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:57.845226 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:57.845249 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 05:03:57.845346 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 05:03:57.845521 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"mutex_wait_us":80,"thread_start_us":599,"threads_started":1}
I20260925 05:03:57.847091 13547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc7d366dd484f779f3b644cfe3101eb" candidate_uuid: "5354129b30144953beced6758ed463e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c8b7b49316b439896cac8f385b46bc5" is_pre_election: true
I20260925 05:03:57.847084 13414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc7d366dd484f779f3b644cfe3101eb" candidate_uuid: "5354129b30144953beced6758ed463e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11fb070ae4d147dd85574dbcc1310515" is_pre_election: true
I20260925 05:03:57.847565 13547 raft_consensus.cc:2454] T bcc7d366dd484f779f3b644cfe3101eb P 3c8b7b49316b439896cac8f385b46bc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5354129b30144953beced6758ed463e8 in term 0.
I20260925 05:03:57.847566 13414 raft_consensus.cc:2454] T bcc7d366dd484f779f3b644cfe3101eb P 11fb070ae4d147dd85574dbcc1310515 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5354129b30144953beced6758ed463e8 in term 0.
I20260925 05:03:57.848183 13214 leader_election.cc:302] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 11fb070ae4d147dd85574dbcc1310515, 3c8b7b49316b439896cac8f385b46bc5, 5354129b30144953beced6758ed463e8; no voters:
I20260925 05:03:57.848621 13618 raft_consensus.cc:2781] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:03:57.848774 13618 raft_consensus.cc:482] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:03:57.848901 13618 raft_consensus.cc:3037] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:03:57.851078 13618 raft_consensus.cc:504] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fb070ae4d147dd85574dbcc1310515" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35113 } } peers { permanent_uuid: "3c8b7b49316b439896cac8f385b46bc5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41937 } } peers { permanent_uuid: "5354129b30144953beced6758ed463e8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45219 } }
I20260925 05:03:57.851732 13618 leader_election.cc:288] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8 [CANDIDATE]: Term 1 election: Requested vote from peers 11fb070ae4d147dd85574dbcc1310515 (127.30.247.2:35113), 3c8b7b49316b439896cac8f385b46bc5 (127.30.247.3:41937)
I20260925 05:03:57.852195 13414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc7d366dd484f779f3b644cfe3101eb" candidate_uuid: "5354129b30144953beced6758ed463e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11fb070ae4d147dd85574dbcc1310515"
I20260925 05:03:57.852296 13547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc7d366dd484f779f3b644cfe3101eb" candidate_uuid: "5354129b30144953beced6758ed463e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c8b7b49316b439896cac8f385b46bc5"
I20260925 05:03:57.852435 13414 raft_consensus.cc:3037] T bcc7d366dd484f779f3b644cfe3101eb P 11fb070ae4d147dd85574dbcc1310515 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:03:57.852519 13547 raft_consensus.cc:3037] T bcc7d366dd484f779f3b644cfe3101eb P 3c8b7b49316b439896cac8f385b46bc5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:03:57.854797 13414 raft_consensus.cc:2454] T bcc7d366dd484f779f3b644cfe3101eb P 11fb070ae4d147dd85574dbcc1310515 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5354129b30144953beced6758ed463e8 in term 1.
I20260925 05:03:57.854880 13547 raft_consensus.cc:2454] T bcc7d366dd484f779f3b644cfe3101eb P 3c8b7b49316b439896cac8f385b46bc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5354129b30144953beced6758ed463e8 in term 1.
I20260925 05:03:57.855229 13215 leader_election.cc:302] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11fb070ae4d147dd85574dbcc1310515, 5354129b30144953beced6758ed463e8; no voters:
I20260925 05:03:57.855551 13618 raft_consensus.cc:2781] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:03:57.855978 13618 raft_consensus.cc:686] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8 [term 1 LEADER]: Becoming Leader. State: Replica: 5354129b30144953beced6758ed463e8, State: Running, Role: LEADER
I20260925 05:03:57.856485 13618 consensus_queue.cc:237] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fb070ae4d147dd85574dbcc1310515" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35113 } } peers { permanent_uuid: "3c8b7b49316b439896cac8f385b46bc5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41937 } } peers { permanent_uuid: "5354129b30144953beced6758ed463e8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45219 } }
I20260925 05:03:57.860965 13135 catalog_manager.cc:5495] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5354129b30144953beced6758ed463e8 (127.30.247.1). New cstate: current_term: 1 leader_uuid: "5354129b30144953beced6758ed463e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fb070ae4d147dd85574dbcc1310515" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c8b7b49316b439896cac8f385b46bc5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5354129b30144953beced6758ed463e8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45219 } health_report { overall_health: HEALTHY } } }
I20260925 05:03:57.913021 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:57.907568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47490 (local address 127.30.247.1:45219)
0925 05:03:57.907806 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:57.907812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:57.907831 (+ 19us) server_negotiation.cc:407] Connection header received
0925 05:03:57.908027 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:57.908031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:57.908122 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:57.908278 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:03:57.908691 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.909420 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.910837 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.911234 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.911365 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:57.911880 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:03:57.912428 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:03:57.912633 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:57.912686 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 05:03:57.912764 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 05:03:57.921311 13414 raft_consensus.cc:1261] T bcc7d366dd484f779f3b644cfe3101eb P 11fb070ae4d147dd85574dbcc1310515 [term 1 FOLLOWER]: Refusing update from remote peer 5354129b30144953beced6758ed463e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:03:57.921339 13547 raft_consensus.cc:1261] T bcc7d366dd484f779f3b644cfe3101eb P 3c8b7b49316b439896cac8f385b46bc5 [term 1 FOLLOWER]: Refusing update from remote peer 5354129b30144953beced6758ed463e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:03:57.922315 13621 consensus_queue.cc:1035] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11fb070ae4d147dd85574dbcc1310515" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:03:57.922648 13618 consensus_queue.cc:1035] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c8b7b49316b439896cac8f385b46bc5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:03:57.934553 13624 mvcc.cc:205] Tried to move back new op lower bound from 7333120564915142656 to 7333120564664389632. Current Snapshot: MvccSnapshot[applied={T|T < 7333120564915142656}]
I20260925 05:03:57.935943 13626 mvcc.cc:205] Tried to move back new op lower bound from 7333120564915142656 to 7333120564664389632. Current Snapshot: MvccSnapshot[applied={T|T < 7333120564915142656}]
I20260925 05:03:57.946141 13136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:03:57.946365 13136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:03:57.949482 13136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee: Sending DeleteTablet for 3 replicas of tablet bcc7d366dd484f779f3b644cfe3101eb
I20260925 05:03:57.950579 13394 tablet_service.cc:1514] Processing DeleteTablet for tablet bcc7d366dd484f779f3b644cfe3101eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:03:57 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:59122
I20260925 05:03:57.950764 13527 tablet_service.cc:1514] Processing DeleteTablet for tablet bcc7d366dd484f779f3b644cfe3101eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:03:57 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:40270
I20260925 05:03:57.950785 13260 tablet_service.cc:1514] Processing DeleteTablet for tablet bcc7d366dd484f779f3b644cfe3101eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:03:57 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:47482
I20260925 05:03:57.951579 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 13197
I20260925 05:03:57.962879 13633 tablet_replica.cc:330] T bcc7d366dd484f779f3b644cfe3101eb P 11fb070ae4d147dd85574dbcc1310515: stopping tablet replica
I20260925 05:03:57.963459 13635 tablet_replica.cc:330] T bcc7d366dd484f779f3b644cfe3101eb P 3c8b7b49316b439896cac8f385b46bc5: stopping tablet replica
I20260925 05:03:57.964016 13633 raft_consensus.cc:2229] T bcc7d366dd484f779f3b644cfe3101eb P 11fb070ae4d147dd85574dbcc1310515 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:03:57.964556 13635 raft_consensus.cc:2229] T bcc7d366dd484f779f3b644cfe3101eb P 3c8b7b49316b439896cac8f385b46bc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:03:57.965101 13633 raft_consensus.cc:2258] T bcc7d366dd484f779f3b644cfe3101eb P 11fb070ae4d147dd85574dbcc1310515 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:03:57.965583 13635 raft_consensus.cc:2258] T bcc7d366dd484f779f3b644cfe3101eb P 3c8b7b49316b439896cac8f385b46bc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:03:57.967875 13635 ts_tablet_manager.cc:1905] T bcc7d366dd484f779f3b644cfe3101eb P 3c8b7b49316b439896cac8f385b46bc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:03:57.969597 13633 ts_tablet_manager.cc:1905] T bcc7d366dd484f779f3b644cfe3101eb P 11fb070ae4d147dd85574dbcc1310515: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:03:57.970863 13634 tablet_replica.cc:330] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8: stopping tablet replica
I20260925 05:03:57.971289 13634 raft_consensus.cc:2229] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:03:57.973474 13634 raft_consensus.cc:2258] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:03:57.975569 13634 ts_tablet_manager.cc:1905] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 05:03:57 dist-test-slave-5n86 krb5kdc[13091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.1@KRBTEST.COM
I20260925 05:03:57.982941 13633 ts_tablet_manager.cc:1918] T bcc7d366dd484f779f3b644cfe3101eb P 11fb070ae4d147dd85574dbcc1310515: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:03:57.983098 13633 log.cc:1196] T bcc7d366dd484f779f3b644cfe3101eb P 11fb070ae4d147dd85574dbcc1310515: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/wal/wals/bcc7d366dd484f779f3b644cfe3101eb
I20260925 05:03:57.983712 13633 ts_tablet_manager.cc:1939] T bcc7d366dd484f779f3b644cfe3101eb P 11fb070ae4d147dd85574dbcc1310515: Deleting consensus metadata
I20260925 05:03:57.983964 13634 ts_tablet_manager.cc:1918] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:03:57.984234 13634 log.cc:1196] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/wal/wals/bcc7d366dd484f779f3b644cfe3101eb
I20260925 05:03:57.984865 13634 ts_tablet_manager.cc:1939] T bcc7d366dd484f779f3b644cfe3101eb P 5354129b30144953beced6758ed463e8: Deleting consensus metadata
I20260925 05:03:57.986755 13122 catalog_manager.cc:4841] TS 5354129b30144953beced6758ed463e8 (127.30.247.1:45219): tablet bcc7d366dd484f779f3b644cfe3101eb (table test-table [id=b49a9874bda641f7a1df5d5085adc191]) successfully deleted
I20260925 05:03:57.987541 13635 ts_tablet_manager.cc:1918] T bcc7d366dd484f779f3b644cfe3101eb P 3c8b7b49316b439896cac8f385b46bc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:03:57.987715 13635 log.cc:1196] T bcc7d366dd484f779f3b644cfe3101eb P 3c8b7b49316b439896cac8f385b46bc5: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/wal/wals/bcc7d366dd484f779f3b644cfe3101eb
I20260925 05:03:57.988364 13635 ts_tablet_manager.cc:1939] T bcc7d366dd484f779f3b644cfe3101eb P 3c8b7b49316b439896cac8f385b46bc5: Deleting consensus metadata
I20260925 05:03:57.989140 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:57.952943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47504 (local address 127.30.247.1:45219)
0925 05:03:57.953152 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:57.953160 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:57.968418 (+ 15258us) server_negotiation.cc:407] Connection header received
0925 05:03:57.968894 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:57.968903 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:57.969024 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:57.969114 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:57.971372 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.972609 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.974558 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:57.975279 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:57.975369 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:57.981114 (+ 5745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:57.981129 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:57.981148 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:57.981194 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:57.985384 (+ 4190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:57.986285 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:57.986293 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:57.986297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:57.986487 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:57.986934 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:57.986939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:57.986942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:57.987369 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:57.987663 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:57.988329 (+ 666us) server_negotiation.cc:299] Negotiation successful
0925 05:03:57.988684 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 05:03:57.989818 13320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:03:57.990727 13121 catalog_manager.cc:4841] TS 3c8b7b49316b439896cac8f385b46bc5 (127.30.247.3:41937): tablet bcc7d366dd484f779f3b644cfe3101eb (table test-table [id=b49a9874bda641f7a1df5d5085adc191]) successfully deleted
I20260925 05:03:57.991843 13122 catalog_manager.cc:4841] TS 11fb070ae4d147dd85574dbcc1310515 (127.30.247.2:35113): tablet bcc7d366dd484f779f3b644cfe3101eb (table test-table [id=b49a9874bda641f7a1df5d5085adc191]) successfully deleted
I20260925 05:03:58.089852 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 13197
I20260925 05:03:58.108217 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 13331
Sep 25 05:03:58 dist-test-slave-5n86 krb5kdc[13091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.2@KRBTEST.COM
I20260925 05:03:58.123831 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:58.109056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59126 (local address 127.30.247.2:35113)
0925 05:03:58.109229 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:58.109233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:58.109251 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:03:58.109468 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:58.109491 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:58.109570 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:58.109664 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:58.111077 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:58.111823 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:58.113336 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:58.113762 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:58.113867 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:58.118029 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:58.118044 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:58.118060 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:58.118097 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:58.121352 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:58.122035 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:58.122042 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:58.122044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:58.122171 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:58.122486 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:58.122490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:58.122491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:58.122876 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:58.123103 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:58.123380 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 05:03:58.123559 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 05:03:58.124572 13454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:03:58.213853 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 13331
I20260925 05:03:58.230909 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 13464
Sep 25 05:03:58 dist-test-slave-5n86 krb5kdc[13091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.3@KRBTEST.COM
I20260925 05:03:58.246356 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:58.231689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40274 (local address 127.30.247.3:41937)
0925 05:03:58.231879 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:58.231884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:58.231902 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:03:58.232058 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:58.232062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:58.232134 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:58.232223 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:58.233547 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:58.234347 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:58.235665 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:58.236047 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:58.236139 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:58.240636 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:58.240661 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:58.240691 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:58.240751 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:58.243619 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:58.244295 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:58.244302 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:58.244305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:58.244437 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:58.244930 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:58.244934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:58.244936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:58.245314 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:58.245558 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:58.245806 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 05:03:58.245963 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 05:03:58.246963 13587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:03:58.334326 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 13464
I20260925 05:03:58.351241 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 13104
I20260925 05:03:58.352573 13166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:03:58.466866 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 13104
I20260925 05:03:58.488147 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:40301
--webserver_interface=127.30.247.62
--webserver_port=35691
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:40301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:03:58.820896 13641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:03:58.821187 13641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:03:58.821280 13641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:03:58.829384 13641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:03:58.829520 13641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:03:58.829576 13641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:03:58.829631 13641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:03:58.829669 13641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:03:58.839906 13641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:40301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:40301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=35691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:03:58.841292 13641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:03:58.842556 13641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:03:58.865204 13648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:58.865227 13650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:58.865331 13647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:03:58.866084 13641 server_base.cc:1034] running on GCE node
I20260925 05:03:58.866964 13641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:03:58.868585 13641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:03:58.869848 13641 hybrid_clock.cc:648] HybridClock initialized: now 1790312638869790 us; error 40 us; skew 500 ppm
I20260925 05:03:58.870289 13641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:03:58 dist-test-slave-5n86 krb5kdc[13091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312638, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:03:58.873986 13641 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:03:58.875583 13641 webserver.cc:460] Webserver started at http://127.30.247.62:35691/ using document root <none> and password file <none>
I20260925 05:03:58.876369 13641 fs_manager.cc:359] Metadata directory not provided
I20260925 05:03:58.876569 13641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:03:58.882496 13641 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 05:03:58.885545 13657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:58.886924 13641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:03:58.887102 13641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "4efda22cdb06437995f8dd492a5bc3ee"
format_stamp: "Formatted at 2026-09-25 05:03:55 on dist-test-slave-5n86"
server_key: "f3e74dd0378b29d0053b0f90d26495d4"
server_key_iv: "fb0c72d25da2803c9dbc3efda0003eb6"
server_key_version: "encryptionkey@0"
I20260925 05:03:58.888152 13641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:03:58.913918 13641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:03:58.918584 13641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:03:58.918866 13641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:03:58.937394 13641 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:40301
I20260925 05:03:58.937414 13708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:40301 every 8 connection(s)
I20260925 05:03:58.938858 13641 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:03:58.940840 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 13641
I20260925 05:03:58.941854 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:45219
--local_ip_for_outbound_sockets=127.30.247.1
--tserver_master_addrs=127.30.247.62:40301
--webserver_port=38093
--webserver_interface=127.30.247.1
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:03:58.948961 13709 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:03:58.952765 13709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee: Bootstrap starting.
I20260925 05:03:58.972065 13709 log.cc:824] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee: Log is configured to *not* fsync() on all Append() calls
I20260925 05:03:58.983464 13709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:03:58.983975 13709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee: Bootstrap complete.
I20260925 05:03:58.993244 13709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efda22cdb06437995f8dd492a5bc3ee" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40301 } }
I20260925 05:03:58.994062 13709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4efda22cdb06437995f8dd492a5bc3ee, State: Initialized, Role: FOLLOWER
I20260925 05:03:58.994622 13709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efda22cdb06437995f8dd492a5bc3ee" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40301 } }
I20260925 05:03:58.994851 13709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:03:58.994983 13709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:03:58.995162 13709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [term 1 FOLLOWER]: Advancing to term 2
I20260925 05:03:58.998814 13709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efda22cdb06437995f8dd492a5bc3ee" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40301 } }
I20260925 05:03:58.999315 13709 leader_election.cc:302] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4efda22cdb06437995f8dd492a5bc3ee; no voters:
I20260925 05:03:58.999953 13709 leader_election.cc:288] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 05:03:59.000098 13713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [term 2 FOLLOWER]: Leader election won for term 2
I20260925 05:03:59.002409 13709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:03:59.006834 13713 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [term 2 LEADER]: Becoming Leader. State: Replica: 4efda22cdb06437995f8dd492a5bc3ee, State: Running, Role: LEADER
I20260925 05:03:59.007468 13713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efda22cdb06437995f8dd492a5bc3ee" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40301 } }
I20260925 05:03:59.014161 13714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4efda22cdb06437995f8dd492a5bc3ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efda22cdb06437995f8dd492a5bc3ee" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40301 } } }
I20260925 05:03:59.014226 13715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4efda22cdb06437995f8dd492a5bc3ee. Latest consensus state: current_term: 2 leader_uuid: "4efda22cdb06437995f8dd492a5bc3ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efda22cdb06437995f8dd492a5bc3ee" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40301 } } }
I20260925 05:03:59.014587 13714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [sys.catalog]: This master's current role is: LEADER
I20260925 05:03:59.014667 13715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [sys.catalog]: This master's current role is: LEADER
I20260925 05:03:59.020781 13727 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 05:03:59.020969 13728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:03:59.028849 13728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:03:59.030210 13728 catalog_manager.cc:1259] Loaded cluster ID: 4a7bb354670f46128f2bb0f015907101
I20260925 05:03:59.030309 13728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:03:59.033434 13728 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 05:03:59.051040 13728 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 05:03:59.060247 13728 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:03:59.061934 13728 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee: Loaded TSK: 0
I20260925 05:03:59.063421 13728 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 05:03:59.277345 13711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:03:59.277611 13711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:03:59.277748 13711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:03:59.284945 13711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:03:59.285046 13711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:03:59.285212 13711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:03:59.294857 13711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:45219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=38093
--tserver_master_addrs=127.30.247.62:40301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:03:59.296125 13711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:03:59.297492 13711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:03:59.312177 13737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:59.312886 13736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:59.312987 13739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:03:59.313695 13711 server_base.cc:1034] running on GCE node
I20260925 05:03:59.314347 13711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:03:59.315443 13711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:03:59.316660 13711 hybrid_clock.cc:648] HybridClock initialized: now 1790312639316597 us; error 55 us; skew 500 ppm
I20260925 05:03:59.317009 13711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:03:59 dist-test-slave-5n86 krb5kdc[13091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312639, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:03:59.320361 13711 init.cc:377] Logged in from keytab as kudu/127.30.247.1@KRBTEST.COM (short username kudu)
I20260925 05:03:59.322069 13711 webserver.cc:460] Webserver started at http://127.30.247.1:38093/ using document root <none> and password file <none>
I20260925 05:03:59.322768 13711 fs_manager.cc:359] Metadata directory not provided
I20260925 05:03:59.322925 13711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:03:59.328600 13711 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 05:03:59.331527 13746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:59.333086 13711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:03:59.333258 13711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "5354129b30144953beced6758ed463e8"
format_stamp: "Formatted at 2026-09-25 05:03:56 on dist-test-slave-5n86"
server_key: "139d48cb069de017c59e2cdf9c5ee6d0"
server_key_iv: "fc4024edeed59fbc74f761b637b6d65e"
server_key_version: "encryptionkey@0"
I20260925 05:03:59.334259 13711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:03:59.353883 13711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:03:59.358443 13711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:03:59.358739 13711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:03:59.359804 13711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:03:59.361261 13711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:03:59.361384 13711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:59.361526 13711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:03:59.361613 13711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:59.396410 13711 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:45219
I20260925 05:03:59.396512 13858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:45219 every 8 connection(s)
I20260925 05:03:59.397907 13711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:03:59.407274 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 13711
I20260925 05:03:59.408003 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:35113
--local_ip_for_outbound_sockets=127.30.247.2
--tserver_master_addrs=127.30.247.62:40301
--webserver_port=34681
--webserver_interface=127.30.247.2
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:03:59 dist-test-slave-5n86 krb5kdc[13091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312639, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:03:59.421859 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:59.400605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:54329 (local address 127.30.247.62:40301)
0925 05:03:59.401172 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:59.401184 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:59.401776 (+ 592us) server_negotiation.cc:407] Connection header received
0925 05:03:59.404007 (+ 2231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:59.404020 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:59.404333 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:59.404700 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:59.406059 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:59.407765 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:59.409547 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:59.410673 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:59.410864 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:59.414420 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:59.414428 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:59.414441 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:59.414482 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:59.417482 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:59.418397 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:59.418404 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:59.418409 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:59.418563 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:59.419037 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:59.419040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:59.419042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:59.419475 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:59.419637 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:59.420624 (+ 987us) server_negotiation.cc:299] Negotiation successful
0925 05:03:59.420928 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":244,"threads_started":1}
I20260925 05:03:59.422286 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:59.400812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.1:54329)
0925 05:03:59.401618 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:59.401662 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:59.403445 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:59.404817 (+ 1372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:59.404824 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:59.405339 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:03:59.405873 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:59.405889 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:59.407949 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:59.407953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:59.409410 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:59.409422 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:59.410925 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:59.410934 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:59.411240 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:59.411946 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:03:59.411976 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:03:59.414161 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:03:59.417622 (+ 3461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:59.417634 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:59.417651 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:59.418258 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:59.418684 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:59.418692 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:59.418698 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:59.418932 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:59.419612 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:03:59.419622 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:03:59.420054 (+ 432us) client_negotiation.cc:769] Sending connection context
0925 05:03:59.420366 (+ 312us) client_negotiation.cc:240] Negotiation successful
0925 05:03:59.420968 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":347,"threads_started":1}
I20260925 05:03:59.425211 13859 heartbeater.cc:347] Connected to a master server at 127.30.247.62:40301
I20260925 05:03:59.425499 13859 heartbeater.cc:464] Registering TS with master...
I20260925 05:03:59.426254 13859 heartbeater.cc:511] Master 127.30.247.62:40301 requested a full tablet report, sending...
I20260925 05:03:59.428779 13674 ts_manager.cc:194] Registered new tserver with Master: 5354129b30144953beced6758ed463e8 (127.30.247.1:45219)
I20260925 05:03:59.431226 13674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.1@KRBTEST.COM'} at 127.30.247.1:54329
W20260925 05:03:59.736192 13863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:03:59.736472 13863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:03:59.736626 13863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:03:59.744097 13863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:03:59.744205 13863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:03:59.744316 13863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:03:59.754086 13863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:35113
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=34681
--tserver_master_addrs=127.30.247.62:40301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:03:59.755330 13863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:03:59.756635 13863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:03:59.769490 13868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:59.769516 13869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:03:59.769537 13871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:03:59.770350 13863 server_base.cc:1034] running on GCE node
I20260925 05:03:59.770897 13863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:03:59.771786 13863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:03:59.772985 13863 hybrid_clock.cc:648] HybridClock initialized: now 1790312639772927 us; error 52 us; skew 500 ppm
I20260925 05:03:59.773324 13863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:03:59 dist-test-slave-5n86 krb5kdc[13091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312639, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:03:59.776468 13863 init.cc:377] Logged in from keytab as kudu/127.30.247.2@KRBTEST.COM (short username kudu)
I20260925 05:03:59.777977 13863 webserver.cc:460] Webserver started at http://127.30.247.2:34681/ using document root <none> and password file <none>
I20260925 05:03:59.778657 13863 fs_manager.cc:359] Metadata directory not provided
I20260925 05:03:59.778810 13863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:03:59.785022 13863 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 05:03:59.787703 13878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:59.789230 13863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:03:59.789397 13863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "11fb070ae4d147dd85574dbcc1310515"
format_stamp: "Formatted at 2026-09-25 05:03:56 on dist-test-slave-5n86"
server_key: "c1fa50450e682937edb3326a05054bc6"
server_key_iv: "69c2d262085337c5f00d30270626e22a"
server_key_version: "encryptionkey@0"
I20260925 05:03:59.790356 13863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:03:59.814797 13863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:03:59.819795 13863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:03:59.820103 13863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:03:59.821210 13863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:03:59.822691 13863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:03:59.822814 13863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:59.822940 13863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:03:59.823024 13863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:03:59.858044 13863 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:35113
I20260925 05:03:59.858127 13990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:35113 every 8 connection(s)
I20260925 05:03:59.859522 13863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:03:59.865126 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 13863
I20260925 05:03:59.865727 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:41937
--local_ip_for_outbound_sockets=127.30.247.3
--tserver_master_addrs=127.30.247.62:40301
--webserver_port=34851
--webserver_interface=127.30.247.3
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:03:59 dist-test-slave-5n86 krb5kdc[13091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312639, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:03:59.880146 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:59.862175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:52425 (local address 127.30.247.62:40301)
0925 05:03:59.862346 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 05:03:59.862350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:03:59.863470 (+ 1120us) server_negotiation.cc:407] Connection header received
0925 05:03:59.865463 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:03:59.865469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:03:59.865563 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:03:59.865674 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:03:59.867261 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:59.867955 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:59.869589 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:03:59.870015 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:03:59.870149 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:59.873948 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:03:59.873953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:03:59.873956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:03:59.873991 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:03:59.876499 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:59.877546 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:59.877551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:59.877553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:59.877696 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:03:59.878268 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:03:59.878273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:03:59.878275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:03:59.878578 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:03:59.878702 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 05:03:59.879686 (+ 984us) server_negotiation.cc:299] Negotiation successful
0925 05:03:59.879875 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 05:03:59.881778 13993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:03:59.862581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.2:52425)
0925 05:03:59.863298 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0925 05:03:59.863336 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 05:03:59.865145 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:03:59.865941 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:03:59.865954 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:03:59.866501 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:03:59.867085 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:59.867103 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:59.868115 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:59.868121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:59.869434 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:03:59.869447 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:03:59.870243 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:03:59.870250 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:03:59.870501 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:03:59.871209 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:03:59.871232 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:03:59.873709 (+ 2477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:03:59.876696 (+ 2987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:59.876708 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:59.876729 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:59.877407 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:59.877813 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:03:59.877822 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:03:59.877828 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:03:59.878091 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:03:59.878717 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:03:59.878728 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:03:59.879224 (+ 496us) client_negotiation.cc:769] Sending connection context
0925 05:03:59.879545 (+ 321us) client_negotiation.cc:240] Negotiation successful
0925 05:03:59.880158 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":276,"threads_started":1}
I20260925 05:03:59.883426 13991 heartbeater.cc:347] Connected to a master server at 127.30.247.62:40301
I20260925 05:03:59.883872 13991 heartbeater.cc:464] Registering TS with master...
I20260925 05:03:59.884719 13991 heartbeater.cc:511] Master 127.30.247.62:40301 requested a full tablet report, sending...
I20260925 05:03:59.886224 13674 ts_manager.cc:194] Registered new tserver with Master: 11fb070ae4d147dd85574dbcc1310515 (127.30.247.2:35113)
I20260925 05:03:59.887068 13674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.2@KRBTEST.COM'} at 127.30.247.2:52425
W20260925 05:04:00.204470 13994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:04:00.204880 13994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:04:00.205068 13994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:04:00.212680 13994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:04:00.212807 13994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:04:00.212981 13994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:04:00.222939 13994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:41937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=34851
--tserver_master_addrs=127.30.247.62:40301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:04:00.224272 13994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:04:00.225687 13994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:04:00.239130 14002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:04:00.239102 14000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:04:00.239373 13999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:04:00.240119 13994 server_base.cc:1034] running on GCE node
I20260925 05:04:00.240782 13994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:04:00.241719 13994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:04:00.242959 13994 hybrid_clock.cc:648] HybridClock initialized: now 1790312640242878 us; error 61 us; skew 500 ppm
I20260925 05:04:00.243302 13994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:04:00 dist-test-slave-5n86 krb5kdc[13091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312640, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:04:00.246874 13994 init.cc:377] Logged in from keytab as kudu/127.30.247.3@KRBTEST.COM (short username kudu)
I20260925 05:04:00.248507 13994 webserver.cc:460] Webserver started at http://127.30.247.3:34851/ using document root <none> and password file <none>
I20260925 05:04:00.249318 13994 fs_manager.cc:359] Metadata directory not provided
I20260925 05:04:00.249444 13994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:04:00.255321 13994 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 05:04:00.258265 14009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:00.259776 13994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:04:00.259924 13994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "3c8b7b49316b439896cac8f385b46bc5"
format_stamp: "Formatted at 2026-09-25 05:03:57 on dist-test-slave-5n86"
server_key: "3f01e680bd34b2de6b6639cb80a695ad"
server_key_iv: "c8a8adde32ffa48dc58f969eacee0ce0"
server_key_version: "encryptionkey@0"
I20260925 05:04:00.260994 13994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:04:00.274438 13994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:04:00.278718 13994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:04:00.278941 13994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:04:00.279862 13994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:04:00.281145 13994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:04:00.281229 13994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:00.281306 13994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:04:00.281358 13994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:00.314327 13994 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:41937
I20260925 05:04:00.314407 14121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:41937 every 8 connection(s)
I20260925 05:04:00.315768 13994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:04:00.320861 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 13994
Sep 25 05:04:00 dist-test-slave-5n86 krb5kdc[13091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312640, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:04:00.337348 14126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:00.321861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51960 (local address 127.30.247.62:40301)
0925 05:04:00.322624 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:00.322628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:00.322642 (+ 14us) server_negotiation.cc:407] Connection header received
0925 05:04:00.322682 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:00.322686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:00.322735 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:00.322821 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:04:00.324584 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.325409 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.327236 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.327713 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.327799 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:00.330807 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:04:00.330811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:04:00.330814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:04:00.330846 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:04:00.334391 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:00.335190 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:00.335194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:00.335197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:00.335343 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:00.335782 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:00.335788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:00.335796 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:00.336100 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:04:00.336354 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:00.336684 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 05:04:00.336917 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"spinlock_wait_cycles":1280,"thread_start_us":555,"threads_started":1}
I20260925 05:04:00.337790 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:00.318297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:48293 (local address 127.30.247.62:40301)
0925 05:04:00.318498 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:00.318503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:00.319653 (+ 1150us) server_negotiation.cc:407] Connection header received
0925 05:04:00.321603 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:00.321609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:00.321717 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:00.321834 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:04:00.323853 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.324691 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.326671 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.327084 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.327207 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:00.331003 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:04:00.331010 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:04:00.331017 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:04:00.331070 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:04:00.334528 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:00.335373 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:00.335380 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:00.335384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:00.335516 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:00.335928 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:00.335935 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:00.335938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:00.336200 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:04:00.336346 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:00.337076 (+ 730us) server_negotiation.cc:299] Negotiation successful
0925 05:04:00.337457 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 05:04:00.339107 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:00.318770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.3:48293)
0925 05:04:00.319470 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:00.319513 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 05:04:00.321329 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:04:00.322101 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:04:00.322114 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:04:00.322786 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:04:00.323646 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:00.323675 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.324844 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.324847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:00.326515 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:00.326529 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.327300 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.327305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:00.327499 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:00.328193 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:04:00.328215 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:04:00.330806 (+ 2591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:04:00.334642 (+ 3836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:04:00.334651 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:04:00.334670 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:04:00.335262 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:04:00.335633 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:04:00.335638 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:04:00.335642 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:04:00.335821 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:04:00.336325 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:04:00.336333 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:04:00.336733 (+ 400us) client_negotiation.cc:769] Sending connection context
0925 05:04:00.336997 (+ 264us) client_negotiation.cc:240] Negotiation successful
0925 05:04:00.337639 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":274,"threads_started":1}
I20260925 05:04:00.341117 14122 heartbeater.cc:347] Connected to a master server at 127.30.247.62:40301
I20260925 05:04:00.341429 14122 heartbeater.cc:464] Registering TS with master...
I20260925 05:04:00.342173 14122 heartbeater.cc:511] Master 127.30.247.62:40301 requested a full tablet report, sending...
I20260925 05:04:00.343577 13674 ts_manager.cc:194] Registered new tserver with Master: 3c8b7b49316b439896cac8f385b46bc5 (127.30.247.3:41937)
I20260925 05:04:00.344417 13674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.3@KRBTEST.COM'} at 127.30.247.3:48293
I20260925 05:04:00.352589 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:04:00.359314 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:00.353496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51974 (local address 127.30.247.62:40301)
0925 05:04:00.353674 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:00.353678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:00.354151 (+ 473us) server_negotiation.cc:407] Connection header received
0925 05:04:00.354328 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:00.354332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:00.354472 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:00.354597 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:04:00.355039 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.355778 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.357315 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.357721 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.357818 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:00.358306 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:04:00.358809 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:04:00.358917 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:00.359037 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 05:04:00.359107 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 05:04:00.360219 13674 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:51974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:04:00.367650 13674 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:04:00.391287 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:00.379572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38588 (local address 127.30.247.2:35113)
0925 05:04:00.382503 (+ 2931us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:00.382515 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:00.382556 (+ 41us) server_negotiation.cc:407] Connection header received
0925 05:04:00.382674 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:00.382685 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:00.382988 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:00.384129 (+ 1141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:04:00.384348 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.385702 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.388781 (+ 3079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.390101 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.390262 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:00.390472 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:00.390755 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 05:04:00.390982 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2708,"thread_start_us":2575,"threads_started":1}
I20260925 05:04:00.391322 14132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:00.379223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:35113 (local address 127.0.0.1:38588)
0925 05:04:00.380254 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:00.380302 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 05:04:00.380495 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:04:00.383262 (+ 2767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:04:00.383275 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:04:00.383334 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:04:00.383837 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:00.383852 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.385843 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.385849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:00.388637 (+ 2788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:00.388652 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.390325 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.390331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:00.390577 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:00.390601 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 05:04:00.390710 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 05:04:00.390918 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":186,"thread_start_us":677,"threads_started":1}
I20260925 05:04:00.391458 14135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:00.380307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46132 (local address 127.30.247.3:41937)
0925 05:04:00.383176 (+ 2869us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:00.383185 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:00.383219 (+ 34us) server_negotiation.cc:407] Connection header received
0925 05:04:00.383332 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:00.383341 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:00.383569 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:00.383780 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:04:00.384375 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.385962 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.388741 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.389663 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.389812 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:00.390057 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:00.390847 (+ 790us) server_negotiation.cc:299] Negotiation successful
0925 05:04:00.391015 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2714,"thread_start_us":2224,"threads_started":1}
I20260925 05:04:00.392685 14136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:00.380745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:45219 (local address 127.0.0.1:45030)
0925 05:04:00.383698 (+ 2953us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:00.383718 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:04:00.383853 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:04:00.384773 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:04:00.384778 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:04:00.384802 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:04:00.385158 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:00.385168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.386915 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.386923 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:00.389604 (+ 2681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:00.389619 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.391735 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.391741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:00.391930 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:00.391962 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 05:04:00.392056 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 05:04:00.392191 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2788,"mutex_wait_us":82,"spinlock_wait_cycles":1408,"thread_start_us":2582,"threads_started":1}
I20260925 05:04:00.392297 14134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:00.379628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:41937 (local address 127.0.0.1:46132)
0925 05:04:00.381117 (+ 1489us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:00.381139 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:04:00.381254 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:04:00.383781 (+ 2527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:04:00.383787 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:04:00.383818 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:04:00.384173 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:00.384185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.386098 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.386111 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:00.388518 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:00.388556 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.390210 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.390216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:00.390447 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:00.390563 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 05:04:00.390696 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 05:04:00.391865 (+ 1169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1361,"mutex_wait_us":1067,"thread_start_us":1206,"threads_started":1}
I20260925 05:04:00.393083 14137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:00.381280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45030 (local address 127.30.247.1:45219)
0925 05:04:00.383161 (+ 1881us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:00.383170 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:00.383997 (+ 827us) server_negotiation.cc:407] Connection header received
0925 05:04:00.384101 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:00.384109 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:00.384315 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:00.384502 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:04:00.385339 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.386719 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.389773 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.391215 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.391354 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:00.391561 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:00.392641 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0925 05:04:00.392830 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1731,"thread_start_us":1584,"threads_started":1}
I20260925 05:04:00.396817 14057 tablet_service.cc:1467] Processing CreateTablet for tablet 8874c89b34984b46b80a78e6edacbf8d (DEFAULT_TABLE table=test-table [id=b641a0a86a5142faada5d41fbf04999c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:04:00.397964 13926 tablet_service.cc:1467] Processing CreateTablet for tablet 8874c89b34984b46b80a78e6edacbf8d (DEFAULT_TABLE table=test-table [id=b641a0a86a5142faada5d41fbf04999c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:04:00.398458 14057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8874c89b34984b46b80a78e6edacbf8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:04:00.399183 13926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8874c89b34984b46b80a78e6edacbf8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:04:00.398729 13794 tablet_service.cc:1467] Processing CreateTablet for tablet 8874c89b34984b46b80a78e6edacbf8d (DEFAULT_TABLE table=test-table [id=b641a0a86a5142faada5d41fbf04999c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:04:00.400027 13794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8874c89b34984b46b80a78e6edacbf8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:04:00.413676 14138 tablet_bootstrap.cc:492] T 8874c89b34984b46b80a78e6edacbf8d P 11fb070ae4d147dd85574dbcc1310515: Bootstrap starting.
I20260925 05:04:00.415742 14139 tablet_bootstrap.cc:492] T 8874c89b34984b46b80a78e6edacbf8d P 3c8b7b49316b439896cac8f385b46bc5: Bootstrap starting.
I20260925 05:04:00.418093 14138 tablet_bootstrap.cc:654] T 8874c89b34984b46b80a78e6edacbf8d P 11fb070ae4d147dd85574dbcc1310515: Neither blocks nor log segments found. Creating new log.
I20260925 05:04:00.419250 14139 tablet_bootstrap.cc:654] T 8874c89b34984b46b80a78e6edacbf8d P 3c8b7b49316b439896cac8f385b46bc5: Neither blocks nor log segments found. Creating new log.
I20260925 05:04:00.419394 14138 log.cc:824] T 8874c89b34984b46b80a78e6edacbf8d P 11fb070ae4d147dd85574dbcc1310515: Log is configured to *not* fsync() on all Append() calls
I20260925 05:04:00.420416 14139 log.cc:824] T 8874c89b34984b46b80a78e6edacbf8d P 3c8b7b49316b439896cac8f385b46bc5: Log is configured to *not* fsync() on all Append() calls
I20260925 05:04:00.422895 14138 tablet_bootstrap.cc:492] T 8874c89b34984b46b80a78e6edacbf8d P 11fb070ae4d147dd85574dbcc1310515: No bootstrap required, opened a new log
I20260925 05:04:00.423182 14138 ts_tablet_manager.cc:1397] T 8874c89b34984b46b80a78e6edacbf8d P 11fb070ae4d147dd85574dbcc1310515: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260925 05:04:00.423605 14139 tablet_bootstrap.cc:492] T 8874c89b34984b46b80a78e6edacbf8d P 3c8b7b49316b439896cac8f385b46bc5: No bootstrap required, opened a new log
I20260925 05:04:00.424175 14139 ts_tablet_manager.cc:1397] T 8874c89b34984b46b80a78e6edacbf8d P 3c8b7b49316b439896cac8f385b46bc5: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 05:04:00.424656 14140 tablet_bootstrap.cc:492] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8: Bootstrap starting.
I20260925 05:04:00.427884 14140 tablet_bootstrap.cc:654] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8: Neither blocks nor log segments found. Creating new log.
I20260925 05:04:00.428890 14140 log.cc:824] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8: Log is configured to *not* fsync() on all Append() calls
I20260925 05:04:00.431434 14140 tablet_bootstrap.cc:492] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8: No bootstrap required, opened a new log
I20260925 05:04:00.431710 14140 ts_tablet_manager.cc:1397] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20260925 05:04:00.434466 13859 heartbeater.cc:503] Master 127.30.247.62:40301 was elected leader, sending a full tablet report...
I20260925 05:04:00.434543 14138 raft_consensus.cc:348] T 8874c89b34984b46b80a78e6edacbf8d P 11fb070ae4d147dd85574dbcc1310515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fb070ae4d147dd85574dbcc1310515" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35113 } } peers { permanent_uuid: "5354129b30144953beced6758ed463e8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45219 } } peers { permanent_uuid: "3c8b7b49316b439896cac8f385b46bc5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41937 } }
I20260925 05:04:00.435004 14138 raft_consensus.cc:374] T 8874c89b34984b46b80a78e6edacbf8d P 11fb070ae4d147dd85574dbcc1310515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:04:00.434768 14139 raft_consensus.cc:348] T 8874c89b34984b46b80a78e6edacbf8d P 3c8b7b49316b439896cac8f385b46bc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fb070ae4d147dd85574dbcc1310515" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35113 } } peers { permanent_uuid: "5354129b30144953beced6758ed463e8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45219 } } peers { permanent_uuid: "3c8b7b49316b439896cac8f385b46bc5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41937 } }
I20260925 05:04:00.435097 14138 raft_consensus.cc:729] T 8874c89b34984b46b80a78e6edacbf8d P 11fb070ae4d147dd85574dbcc1310515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11fb070ae4d147dd85574dbcc1310515, State: Initialized, Role: FOLLOWER
I20260925 05:04:00.435158 14139 raft_consensus.cc:374] T 8874c89b34984b46b80a78e6edacbf8d P 3c8b7b49316b439896cac8f385b46bc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:04:00.435240 14139 raft_consensus.cc:729] T 8874c89b34984b46b80a78e6edacbf8d P 3c8b7b49316b439896cac8f385b46bc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c8b7b49316b439896cac8f385b46bc5, State: Initialized, Role: FOLLOWER
I20260925 05:04:00.435835 14139 consensus_queue.cc:260] T 8874c89b34984b46b80a78e6edacbf8d P 3c8b7b49316b439896cac8f385b46bc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fb070ae4d147dd85574dbcc1310515" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35113 } } peers { permanent_uuid: "5354129b30144953beced6758ed463e8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45219 } } peers { permanent_uuid: "3c8b7b49316b439896cac8f385b46bc5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41937 } }
I20260925 05:04:00.435837 14138 consensus_queue.cc:260] T 8874c89b34984b46b80a78e6edacbf8d P 11fb070ae4d147dd85574dbcc1310515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fb070ae4d147dd85574dbcc1310515" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35113 } } peers { permanent_uuid: "5354129b30144953beced6758ed463e8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45219 } } peers { permanent_uuid: "3c8b7b49316b439896cac8f385b46bc5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41937 } }
I20260925 05:04:00.442085 14138 ts_tablet_manager.cc:1428] T 8874c89b34984b46b80a78e6edacbf8d P 11fb070ae4d147dd85574dbcc1310515: Time spent starting tablet: real 0.019s user 0.015s sys 0.003s
I20260925 05:04:00.442551 13991 heartbeater.cc:503] Master 127.30.247.62:40301 was elected leader, sending a full tablet report...
I20260925 05:04:00.442562 14140 raft_consensus.cc:348] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fb070ae4d147dd85574dbcc1310515" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35113 } } peers { permanent_uuid: "5354129b30144953beced6758ed463e8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45219 } } peers { permanent_uuid: "3c8b7b49316b439896cac8f385b46bc5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41937 } }
I20260925 05:04:00.443050 14140 raft_consensus.cc:374] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:04:00.443151 14140 raft_consensus.cc:729] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5354129b30144953beced6758ed463e8, State: Initialized, Role: FOLLOWER
I20260925 05:04:00.443775 14140 consensus_queue.cc:260] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fb070ae4d147dd85574dbcc1310515" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35113 } } peers { permanent_uuid: "5354129b30144953beced6758ed463e8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45219 } } peers { permanent_uuid: "3c8b7b49316b439896cac8f385b46bc5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41937 } }
I20260925 05:04:00.444512 14122 heartbeater.cc:503] Master 127.30.247.62:40301 was elected leader, sending a full tablet report...
I20260925 05:04:00.446432 14139 ts_tablet_manager.cc:1428] T 8874c89b34984b46b80a78e6edacbf8d P 3c8b7b49316b439896cac8f385b46bc5: Time spent starting tablet: real 0.022s user 0.012s sys 0.008s
I20260925 05:04:00.446741 14140 ts_tablet_manager.cc:1428] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260925 05:04:00.486426 14146 raft_consensus.cc:482] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:04:00.486721 14146 raft_consensus.cc:504] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fb070ae4d147dd85574dbcc1310515" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35113 } } peers { permanent_uuid: "5354129b30144953beced6758ed463e8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45219 } } peers { permanent_uuid: "3c8b7b49316b439896cac8f385b46bc5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41937 } }
I20260925 05:04:00.488600 14146 leader_election.cc:288] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11fb070ae4d147dd85574dbcc1310515 (127.30.247.2:35113), 3c8b7b49316b439896cac8f385b46bc5 (127.30.247.3:41937)
I20260925 05:04:00.495682 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:00.489202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:50529 (local address 127.30.247.2:35113)
0925 05:04:00.489441 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:00.489451 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:00.490131 (+ 680us) server_negotiation.cc:407] Connection header received
0925 05:04:00.490352 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:00.490356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:00.490416 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:00.490518 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:04:00.491110 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.491850 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.493926 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.494768 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.494938 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:00.495021 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:00.495296 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 05:04:00.495421 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 05:04:00.495878 14135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:00.489153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:57395 (local address 127.30.247.3:41937)
0925 05:04:00.489396 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:00.489405 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:00.490353 (+ 948us) server_negotiation.cc:407] Connection header received
0925 05:04:00.490552 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:00.490556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:00.490622 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:00.490715 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:04:00.491340 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.492115 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.494083 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.494943 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.495119 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:00.495199 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:00.495488 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 05:04:00.495625 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 05:04:00.495981 14148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:00.489065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:41937 (local address 127.30.247.1:57395)
0925 05:04:00.490245 (+ 1180us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:00.490258 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:04:00.490400 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:04:00.490736 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:04:00.490740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:04:00.490764 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:04:00.491162 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:00.491170 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.492262 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.492267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:00.493929 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:00.493945 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.495167 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.495174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:00.495335 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:00.495358 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 05:04:00.495461 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 05:04:00.495586 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1082,"mutex_wait_us":53,"thread_start_us":851,"threads_started":1}
I20260925 05:04:00.496227 14147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:00.489021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:35113 (local address 127.30.247.1:50529)
0925 05:04:00.490016 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:00.490033 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:04:00.490206 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:04:00.490541 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:04:00.490546 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:04:00.490571 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:04:00.490941 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:00.490953 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.491971 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.491979 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:00.493753 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:00.493765 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.494964 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.494970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:00.495140 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:00.495167 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 05:04:00.495270 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 05:04:00.495396 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"thread_start_us":611,"threads_started":1}
I20260925 05:04:00.496965 14077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8874c89b34984b46b80a78e6edacbf8d" candidate_uuid: "5354129b30144953beced6758ed463e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c8b7b49316b439896cac8f385b46bc5" is_pre_election: true
I20260925 05:04:00.497128 13946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8874c89b34984b46b80a78e6edacbf8d" candidate_uuid: "5354129b30144953beced6758ed463e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11fb070ae4d147dd85574dbcc1310515" is_pre_election: true
I20260925 05:04:00.497486 14077 raft_consensus.cc:2454] T 8874c89b34984b46b80a78e6edacbf8d P 3c8b7b49316b439896cac8f385b46bc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5354129b30144953beced6758ed463e8 in term 0.
I20260925 05:04:00.497630 13946 raft_consensus.cc:2454] T 8874c89b34984b46b80a78e6edacbf8d P 11fb070ae4d147dd85574dbcc1310515 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5354129b30144953beced6758ed463e8 in term 0.
I20260925 05:04:00.498159 13748 leader_election.cc:302] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c8b7b49316b439896cac8f385b46bc5, 5354129b30144953beced6758ed463e8; no voters:
I20260925 05:04:00.498571 14146 raft_consensus.cc:2781] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:04:00.498720 14146 raft_consensus.cc:482] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:04:00.498832 14146 raft_consensus.cc:3037] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:04:00.501073 14146 raft_consensus.cc:504] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fb070ae4d147dd85574dbcc1310515" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35113 } } peers { permanent_uuid: "5354129b30144953beced6758ed463e8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45219 } } peers { permanent_uuid: "3c8b7b49316b439896cac8f385b46bc5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41937 } }
I20260925 05:04:00.501631 14146 leader_election.cc:288] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8 [CANDIDATE]: Term 1 election: Requested vote from peers 11fb070ae4d147dd85574dbcc1310515 (127.30.247.2:35113), 3c8b7b49316b439896cac8f385b46bc5 (127.30.247.3:41937)
I20260925 05:04:00.502025 13946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8874c89b34984b46b80a78e6edacbf8d" candidate_uuid: "5354129b30144953beced6758ed463e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11fb070ae4d147dd85574dbcc1310515"
I20260925 05:04:00.502120 14077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8874c89b34984b46b80a78e6edacbf8d" candidate_uuid: "5354129b30144953beced6758ed463e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c8b7b49316b439896cac8f385b46bc5"
I20260925 05:04:00.502230 13946 raft_consensus.cc:3037] T 8874c89b34984b46b80a78e6edacbf8d P 11fb070ae4d147dd85574dbcc1310515 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:04:00.502357 14077 raft_consensus.cc:3037] T 8874c89b34984b46b80a78e6edacbf8d P 3c8b7b49316b439896cac8f385b46bc5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:04:00.504652 13946 raft_consensus.cc:2454] T 8874c89b34984b46b80a78e6edacbf8d P 11fb070ae4d147dd85574dbcc1310515 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5354129b30144953beced6758ed463e8 in term 1.
I20260925 05:04:00.504845 14077 raft_consensus.cc:2454] T 8874c89b34984b46b80a78e6edacbf8d P 3c8b7b49316b439896cac8f385b46bc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5354129b30144953beced6758ed463e8 in term 1.
I20260925 05:04:00.505060 13749 leader_election.cc:302] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11fb070ae4d147dd85574dbcc1310515, 5354129b30144953beced6758ed463e8; no voters:
I20260925 05:04:00.505385 14146 raft_consensus.cc:2781] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:04:00.505827 14146 raft_consensus.cc:686] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8 [term 1 LEADER]: Becoming Leader. State: Replica: 5354129b30144953beced6758ed463e8, State: Running, Role: LEADER
I20260925 05:04:00.506291 14146 consensus_queue.cc:237] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fb070ae4d147dd85574dbcc1310515" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35113 } } peers { permanent_uuid: "5354129b30144953beced6758ed463e8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45219 } } peers { permanent_uuid: "3c8b7b49316b439896cac8f385b46bc5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41937 } }
I20260925 05:04:00.511269 13674 catalog_manager.cc:5495] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5354129b30144953beced6758ed463e8 (127.30.247.1). New cstate: current_term: 1 leader_uuid: "5354129b30144953beced6758ed463e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fb070ae4d147dd85574dbcc1310515" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5354129b30144953beced6758ed463e8" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c8b7b49316b439896cac8f385b46bc5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41937 } health_report { overall_health: UNKNOWN } } }
I20260925 05:04:00.555827 14137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:00.550297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45038 (local address 127.30.247.1:45219)
0925 05:04:00.550557 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:00.550564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:00.550626 (+ 62us) server_negotiation.cc:407] Connection header received
0925 05:04:00.550869 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:00.550874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:00.550953 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:00.551123 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:04:00.551476 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.552204 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.553687 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.554110 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.554243 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:00.554750 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:04:00.555256 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:04:00.555410 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:00.555471 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 05:04:00.555551 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 05:04:00.564064 13946 raft_consensus.cc:1261] T 8874c89b34984b46b80a78e6edacbf8d P 11fb070ae4d147dd85574dbcc1310515 [term 1 FOLLOWER]: Refusing update from remote peer 5354129b30144953beced6758ed463e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:04:00.564061 14077 raft_consensus.cc:1261] T 8874c89b34984b46b80a78e6edacbf8d P 3c8b7b49316b439896cac8f385b46bc5 [term 1 FOLLOWER]: Refusing update from remote peer 5354129b30144953beced6758ed463e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:04:00.565137 14149 consensus_queue.cc:1035] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11fb070ae4d147dd85574dbcc1310515" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:04:00.565593 14146 consensus_queue.cc:1035] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c8b7b49316b439896cac8f385b46bc5" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 05:04:00.567361 14123 tablet.cc:2369] T 8874c89b34984b46b80a78e6edacbf8d P 3c8b7b49316b439896cac8f385b46bc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:04:00.577329 14151 mvcc.cc:205] Tried to move back new op lower bound from 7333120575740096512 to 7333120575518138368. Current Snapshot: MvccSnapshot[applied={T|T < 7333120575740096512}]
I20260925 05:04:00.591723 13674 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:51974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:04:00.591982 13674 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:51974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:04:00.595511 13674 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee: Sending DeleteTablet for 3 replicas of tablet 8874c89b34984b46b80a78e6edacbf8d
I20260925 05:04:00.596855 13794 tablet_service.cc:1514] Processing DeleteTablet for tablet 8874c89b34984b46b80a78e6edacbf8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:04:00 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:45030
I20260925 05:04:00.596845 14057 tablet_service.cc:1514] Processing DeleteTablet for tablet 8874c89b34984b46b80a78e6edacbf8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:04:00 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:46132
I20260925 05:04:00.597803 14162 tablet_replica.cc:330] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8: stopping tablet replica
I20260925 05:04:00.597900 13926 tablet_service.cc:1514] Processing DeleteTablet for tablet 8874c89b34984b46b80a78e6edacbf8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:04:00 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:38588
I20260925 05:04:00.598237 14162 raft_consensus.cc:2229] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:04:00.598901 14162 raft_consensus.cc:2258] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:04:00.599114 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 13711
I20260925 05:04:00.599654 14161 tablet_replica.cc:330] T 8874c89b34984b46b80a78e6edacbf8d P 3c8b7b49316b439896cac8f385b46bc5: stopping tablet replica
I20260925 05:04:00.601301 14161 raft_consensus.cc:2229] T 8874c89b34984b46b80a78e6edacbf8d P 3c8b7b49316b439896cac8f385b46bc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:04:00.601795 14161 raft_consensus.cc:2258] T 8874c89b34984b46b80a78e6edacbf8d P 3c8b7b49316b439896cac8f385b46bc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:04:00.602340 14162 ts_tablet_manager.cc:1905] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:04:00.603152 14163 tablet_replica.cc:330] T 8874c89b34984b46b80a78e6edacbf8d P 11fb070ae4d147dd85574dbcc1310515: stopping tablet replica
I20260925 05:04:00.603647 14163 raft_consensus.cc:2229] T 8874c89b34984b46b80a78e6edacbf8d P 11fb070ae4d147dd85574dbcc1310515 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:04:00.605502 14161 ts_tablet_manager.cc:1905] T 8874c89b34984b46b80a78e6edacbf8d P 3c8b7b49316b439896cac8f385b46bc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:04:00.606221 14163 raft_consensus.cc:2258] T 8874c89b34984b46b80a78e6edacbf8d P 11fb070ae4d147dd85574dbcc1310515 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:04:00.609103 14163 ts_tablet_manager.cc:1905] T 8874c89b34984b46b80a78e6edacbf8d P 11fb070ae4d147dd85574dbcc1310515: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:04:00.610000 14162 ts_tablet_manager.cc:1918] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:04:00.610184 14162 log.cc:1196] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/wal/wals/8874c89b34984b46b80a78e6edacbf8d
I20260925 05:04:00.610841 14162 ts_tablet_manager.cc:1939] T 8874c89b34984b46b80a78e6edacbf8d P 5354129b30144953beced6758ed463e8: Deleting consensus metadata
I20260925 05:04:00.612730 13660 catalog_manager.cc:4841] TS 5354129b30144953beced6758ed463e8 (127.30.247.1:45219): tablet 8874c89b34984b46b80a78e6edacbf8d (table test-table [id=b641a0a86a5142faada5d41fbf04999c]) successfully deleted
I20260925 05:04:00.615935 14163 ts_tablet_manager.cc:1918] T 8874c89b34984b46b80a78e6edacbf8d P 11fb070ae4d147dd85574dbcc1310515: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:04:00.616169 14163 log.cc:1196] T 8874c89b34984b46b80a78e6edacbf8d P 11fb070ae4d147dd85574dbcc1310515: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/wal/wals/8874c89b34984b46b80a78e6edacbf8d
I20260925 05:04:00.616714 14163 ts_tablet_manager.cc:1939] T 8874c89b34984b46b80a78e6edacbf8d P 11fb070ae4d147dd85574dbcc1310515: Deleting consensus metadata
I20260925 05:04:00.617899 14137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:00.600036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45046 (local address 127.30.247.1:45219)
0925 05:04:00.600202 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:00.600209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:00.600278 (+ 69us) server_negotiation.cc:407] Connection header received
0925 05:04:00.600778 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:00.600783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:00.600864 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:00.600977 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:04:00.602839 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.604093 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.606047 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.606745 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.606880 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:00.610267 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:04:00.610280 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:04:00.610300 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:04:00.610348 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:04:00.614504 (+ 4156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:00.615373 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:00.615382 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:00.615386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:00.615585 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:00.616066 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:00.616071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:00.616077 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:00.616509 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:04:00.616881 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:00.617250 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 05:04:00.617481 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 05:04:00.618495 13660 catalog_manager.cc:4841] TS 11fb070ae4d147dd85574dbcc1310515 (127.30.247.2:35113): tablet 8874c89b34984b46b80a78e6edacbf8d (table test-table [id=b641a0a86a5142faada5d41fbf04999c]) successfully deleted
I20260925 05:04:00.618621 13854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:04:00.620731 14161 ts_tablet_manager.cc:1918] T 8874c89b34984b46b80a78e6edacbf8d P 3c8b7b49316b439896cac8f385b46bc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:04:00.620900 14161 log.cc:1196] T 8874c89b34984b46b80a78e6edacbf8d P 3c8b7b49316b439896cac8f385b46bc5: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/wal/wals/8874c89b34984b46b80a78e6edacbf8d
I20260925 05:04:00.621456 14161 ts_tablet_manager.cc:1939] T 8874c89b34984b46b80a78e6edacbf8d P 3c8b7b49316b439896cac8f385b46bc5: Deleting consensus metadata
I20260925 05:04:00.623215 13659 catalog_manager.cc:4841] TS 3c8b7b49316b439896cac8f385b46bc5 (127.30.247.3:41937): tablet 8874c89b34984b46b80a78e6edacbf8d (table test-table [id=b641a0a86a5142faada5d41fbf04999c]) successfully deleted
I20260925 05:04:00.717408 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 13711
I20260925 05:04:00.737378 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 13863
I20260925 05:04:00.761970 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:00.738296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38596 (local address 127.30.247.2:35113)
0925 05:04:00.738498 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:00.738505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:00.738523 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:04:00.738685 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:00.738691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:00.738773 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:00.738880 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:04:00.740338 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.741185 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.742467 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.742895 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.743035 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:00.755502 (+ 12467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:04:00.755518 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:04:00.755534 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:04:00.755575 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:04:00.758375 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:00.759556 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:00.759570 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:00.759572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:00.759733 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:00.760191 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:00.760196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:00.760245 (+ 49us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:00.760770 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:04:00.761085 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:00.761392 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 05:04:00.761625 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 05:04:00.762696 13986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:04:00.856160 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 13863
I20260925 05:04:00.873338 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 13994
I20260925 05:04:00.887292 14135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:00.874175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46144 (local address 127.30.247.3:41937)
0925 05:04:00.874386 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:00.874392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:00.874513 (+ 121us) server_negotiation.cc:407] Connection header received
0925 05:04:00.874744 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:00.874749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:00.874827 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:00.874951 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:04:00.876250 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.877051 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.878390 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:00.878790 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:00.878937 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:00.881702 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:04:00.881718 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:04:00.881733 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:04:00.881769 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:04:00.884409 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:00.885187 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:00.885195 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:00.885197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:00.885335 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:00.885721 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:00.885727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:00.885731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:00.886100 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:04:00.886394 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:00.886734 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 05:04:00.886962 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 05:04:00.888006 14117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:04:00.978794 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 13994
I20260925 05:04:00.996351 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 13641
I20260925 05:04:00.997539 13704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:04:01.111389 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 13641
I20260925 05:04:01.133260 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:40301
--webserver_interface=127.30.247.62
--webserver_port=35691
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:40301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:04:01.461516 14168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:04:01.461860 14168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:04:01.462033 14168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:04:01.469190 14168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:04:01.469323 14168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:04:01.469437 14168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:04:01.469547 14168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:04:01.469627 14168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:04:01.479285 14168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:40301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:40301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=35691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:04:01.480674 14168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:04:01.481980 14168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:04:01.493440 14173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:04:01.493641 14174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:04:01.493669 14176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:04:01.494958 14168 server_base.cc:1034] running on GCE node
I20260925 05:04:01.495841 14168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:04:01.497367 14168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:04:01.498672 14168 hybrid_clock.cc:648] HybridClock initialized: now 1790312641498627 us; error 67 us; skew 500 ppm
I20260925 05:04:01.499060 14168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:04:01 dist-test-slave-5n86 krb5kdc[13091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312641, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:04:01.502300 14168 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:04:01.503769 14168 webserver.cc:460] Webserver started at http://127.30.247.62:35691/ using document root <none> and password file <none>
I20260925 05:04:01.504468 14168 fs_manager.cc:359] Metadata directory not provided
I20260925 05:04:01.504634 14168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:04:01.510026 14168 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 05:04:01.513244 14183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:01.514648 14168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:04:01.514854 14168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "4efda22cdb06437995f8dd492a5bc3ee"
format_stamp: "Formatted at 2026-09-25 05:03:55 on dist-test-slave-5n86"
server_key: "f3e74dd0378b29d0053b0f90d26495d4"
server_key_iv: "fb0c72d25da2803c9dbc3efda0003eb6"
server_key_version: "encryptionkey@0"
I20260925 05:04:01.515838 14168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:04:01.533622 14168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:04:01.537998 14168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:04:01.538295 14168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:04:01.557228 14168 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:40301
I20260925 05:04:01.557250 14234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:40301 every 8 connection(s)
I20260925 05:04:01.558786 14168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:04:01.565663 14235 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:04:01.566877 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 14168
I20260925 05:04:01.567732 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:45219
--local_ip_for_outbound_sockets=127.30.247.1
--tserver_master_addrs=127.30.247.62:40301
--webserver_port=38093
--webserver_interface=127.30.247.1
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:04:01.569128 14235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee: Bootstrap starting.
I20260925 05:04:01.596575 14235 log.cc:824] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee: Log is configured to *not* fsync() on all Append() calls
I20260925 05:04:01.616925 14235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:04:01.617550 14235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee: Bootstrap complete.
I20260925 05:04:01.631466 14235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efda22cdb06437995f8dd492a5bc3ee" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40301 } }
I20260925 05:04:01.633072 14235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4efda22cdb06437995f8dd492a5bc3ee, State: Initialized, Role: FOLLOWER
I20260925 05:04:01.633920 14235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efda22cdb06437995f8dd492a5bc3ee" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40301 } }
I20260925 05:04:01.634250 14235 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:04:01.634402 14235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:04:01.634629 14235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [term 2 FOLLOWER]: Advancing to term 3
I20260925 05:04:01.640525 14235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efda22cdb06437995f8dd492a5bc3ee" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40301 } }
I20260925 05:04:01.641202 14235 leader_election.cc:302] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4efda22cdb06437995f8dd492a5bc3ee; no voters:
I20260925 05:04:01.641891 14235 leader_election.cc:288] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 05:04:01.642046 14239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [term 3 FOLLOWER]: Leader election won for term 3
I20260925 05:04:01.643429 14239 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [term 3 LEADER]: Becoming Leader. State: Replica: 4efda22cdb06437995f8dd492a5bc3ee, State: Running, Role: LEADER
I20260925 05:04:01.643970 14235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:04:01.644081 14239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efda22cdb06437995f8dd492a5bc3ee" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40301 } }
I20260925 05:04:01.646524 14241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4efda22cdb06437995f8dd492a5bc3ee. Latest consensus state: current_term: 3 leader_uuid: "4efda22cdb06437995f8dd492a5bc3ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efda22cdb06437995f8dd492a5bc3ee" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40301 } } }
I20260925 05:04:01.646951 14241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [sys.catalog]: This master's current role is: LEADER
I20260925 05:04:01.647284 14240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4efda22cdb06437995f8dd492a5bc3ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efda22cdb06437995f8dd492a5bc3ee" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40301 } } }
I20260925 05:04:01.647598 14240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4efda22cdb06437995f8dd492a5bc3ee [sys.catalog]: This master's current role is: LEADER
I20260925 05:04:01.657250 14248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:04:01.665464 14248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:04:01.667272 14248 catalog_manager.cc:1259] Loaded cluster ID: 4a7bb354670f46128f2bb0f015907101
I20260925 05:04:01.667380 14248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 05:04:01.670125 14248 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f55f225c9ad google::LogMessage::Fail() at ??:0
@ 0x7f55f225f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f55f225c4de google::LogMessage::Flush() at ??:0
@ 0x7f55f225da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5607c8f512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f5607c8c976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5607d14610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f5607d14439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f55f4ea8345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f55f4eab91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f55f4eab799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f55f4e86eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f55f81606db start_thread at ??:0
@ 0x7f55f10f661f clone at ??:0
W20260925 05:04:01.975380 14237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:04:01.975637 14237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:04:01.975792 14237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:04:01.982788 14237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:04:01.982877 14237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:04:01.983068 14237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:04:01.992237 14237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:45219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=38093
--tserver_master_addrs=127.30.247.62:40301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:04:01.993387 14237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:04:01.994625 14237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:04:02.006712 14276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:04:02.006783 14277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:04:02.006760 14279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:04:02.007725 14237 server_base.cc:1034] running on GCE node
I20260925 05:04:02.008272 14237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:04:02.009171 14237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:04:02.010355 14237 hybrid_clock.cc:648] HybridClock initialized: now 1790312642010311 us; error 35 us; skew 500 ppm
I20260925 05:04:02.010694 14237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:04:02 dist-test-slave-5n86 krb5kdc[13091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312642, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:04:02.013721 14237 init.cc:377] Logged in from keytab as kudu/127.30.247.1@KRBTEST.COM (short username kudu)
I20260925 05:04:02.015009 14237 webserver.cc:460] Webserver started at http://127.30.247.1:38093/ using document root <none> and password file <none>
I20260925 05:04:02.015631 14237 fs_manager.cc:359] Metadata directory not provided
I20260925 05:04:02.015766 14237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:04:02.020787 14237 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:04:02.024812 14286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:02.026000 14237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:04:02.026158 14237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "5354129b30144953beced6758ed463e8"
format_stamp: "Formatted at 2026-09-25 05:03:56 on dist-test-slave-5n86"
server_key: "139d48cb069de017c59e2cdf9c5ee6d0"
server_key_iv: "fc4024edeed59fbc74f761b637b6d65e"
server_key_version: "encryptionkey@0"
I20260925 05:04:02.027071 14237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:04:02.044476 14237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:04:02.048190 14237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:04:02.048435 14237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:04:02.049350 14237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:04:02.050580 14237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:04:02.050699 14237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:02.050829 14237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:04:02.050916 14237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:02.083244 14237 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:45219
I20260925 05:04:02.083321 14398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:45219 every 8 connection(s)
I20260925 05:04:02.084684 14237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:04:02.088704 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 14237
I20260925 05:04:02.089470 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:35113
--local_ip_for_outbound_sockets=127.30.247.2
--tserver_master_addrs=127.30.247.62:40301
--webserver_port=34681
--webserver_interface=127.30.247.2
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:04:02.089744 14401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:02.087445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.1:39529)
0925 05:04:02.088258 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:02.088607 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":295,"threads_started":1}
W20260925 05:04:02.090435 14399 heartbeater.cc:644] Failed to heartbeat to 127.30.247.62:40301 (0 consecutive failures): Network error: Failed to ping master at 127.30.247.62:40301: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
I20260925 05:04:02.091305 14401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:02.090892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.1:41259)
0925 05:04:02.091065 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:02.091160 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 05:04:02.092254 14401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:02.091836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.1:56329)
0925 05:04:02.091999 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:02.092097 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 05:04:02.092586 14399 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 05:04:02.399693 14402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:04:02.399964 14402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:04:02.400131 14402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:04:02.407020 14402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:04:02.407158 14402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:04:02.407320 14402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:04:02.416486 14402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:35113
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=34681
--tserver_master_addrs=127.30.247.62:40301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:04:02.417629 14402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:04:02.418805 14402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:04:02.431581 14410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:04:02.431734 14407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:04:02.431885 14408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:04:02.432816 14402 server_base.cc:1034] running on GCE node
I20260925 05:04:02.433336 14402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:04:02.434190 14402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:04:02.435417 14402 hybrid_clock.cc:648] HybridClock initialized: now 1790312642435374 us; error 34 us; skew 500 ppm
I20260925 05:04:02.435752 14402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:04:02 dist-test-slave-5n86 krb5kdc[13091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312642, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:04:02.444619 14402 init.cc:377] Logged in from keytab as kudu/127.30.247.2@KRBTEST.COM (short username kudu)
I20260925 05:04:02.445964 14402 webserver.cc:460] Webserver started at http://127.30.247.2:34681/ using document root <none> and password file <none>
I20260925 05:04:02.446624 14402 fs_manager.cc:359] Metadata directory not provided
I20260925 05:04:02.446756 14402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:04:02.451850 14402 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:04:02.454455 14417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:02.455709 14402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:04:02.455883 14402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "11fb070ae4d147dd85574dbcc1310515"
format_stamp: "Formatted at 2026-09-25 05:03:56 on dist-test-slave-5n86"
server_key: "c1fa50450e682937edb3326a05054bc6"
server_key_iv: "69c2d262085337c5f00d30270626e22a"
server_key_version: "encryptionkey@0"
I20260925 05:04:02.456867 14402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:04:02.478065 14402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:04:02.481715 14402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:04:02.481974 14402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:04:02.482864 14402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:04:02.484076 14402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:04:02.484192 14402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:02.484319 14402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:04:02.484406 14402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:02.516700 14402 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:35113
I20260925 05:04:02.516772 14529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:35113 every 8 connection(s)
I20260925 05:04:02.518029 14402 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:04:02.519822 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 14402
I20260925 05:04:02.520365 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:41937
--local_ip_for_outbound_sockets=127.30.247.3
--tserver_master_addrs=127.30.247.62:40301
--webserver_port=34851
--webserver_interface=127.30.247.3
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:04:02.523067 14532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:02.520904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.2:45615)
0925 05:04:02.521709 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:02.521969 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":288,"threads_started":1}
W20260925 05:04:02.523732 14530 heartbeater.cc:644] Failed to heartbeat to 127.30.247.62:40301 (0 consecutive failures): Network error: Failed to ping master at 127.30.247.62:40301: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
I20260925 05:04:02.524591 14532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:02.524243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.2:36105)
0925 05:04:02.524381 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:02.524447 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 05:04:02.525447 14532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:02.525132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.2:53691)
0925 05:04:02.525267 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:02.525346 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 05:04:02.525686 14530 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 05:04:02.844691 14533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:04:02.844931 14533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:04:02.845082 14533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:04:02.851876 14533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:04:02.851964 14533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:04:02.852174 14533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:04:02.861477 14533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:41937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=34851
--tserver_master_addrs=127.30.247.62:40301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:04:02.862650 14533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:04:02.863953 14533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:04:02.876519 14539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:04:02.876471 14541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:04:02.876603 14538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:04:02.881731 14533 server_base.cc:1034] running on GCE node
I20260925 05:04:02.882266 14533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:04:02.883106 14533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:04:02.884295 14533 hybrid_clock.cc:648] HybridClock initialized: now 1790312642884259 us; error 34 us; skew 500 ppm
I20260925 05:04:02.884641 14533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:04:02 dist-test-slave-5n86 krb5kdc[13091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312642, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:04:02.887897 14533 init.cc:377] Logged in from keytab as kudu/127.30.247.3@KRBTEST.COM (short username kudu)
I20260925 05:04:02.889359 14533 webserver.cc:460] Webserver started at http://127.30.247.3:34851/ using document root <none> and password file <none>
I20260925 05:04:02.890033 14533 fs_manager.cc:359] Metadata directory not provided
I20260925 05:04:02.890159 14533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:04:02.895123 14533 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 05:04:02.897730 14548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:02.898965 14533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 05:04:02.899133 14533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "3c8b7b49316b439896cac8f385b46bc5"
format_stamp: "Formatted at 2026-09-25 05:03:57 on dist-test-slave-5n86"
server_key: "3f01e680bd34b2de6b6639cb80a695ad"
server_key_iv: "c8a8adde32ffa48dc58f969eacee0ce0"
server_key_version: "encryptionkey@0"
I20260925 05:04:02.900048 14533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:04:02.921375 14533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:04:02.925033 14533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:04:02.925275 14533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:04:02.926128 14533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:04:02.927343 14533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:04:02.927448 14533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:02.927582 14533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:04:02.927660 14533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:02.960108 14533 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:41937
I20260925 05:04:02.960183 14660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:41937 every 8 connection(s)
I20260925 05:04:02.961570 14533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:04:02.965986 14663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:02.964246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.3:46613)
0925 05:04:02.964951 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:02.965176 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":310,"threads_started":1}
W20260925 05:04:02.966586 14661 heartbeater.cc:644] Failed to heartbeat to 127.30.247.62:40301 (0 consecutive failures): Network error: Failed to ping master at 127.30.247.62:40301: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
I20260925 05:04:02.967254 14663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:02.966972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.3:49989)
0925 05:04:02.967098 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:02.967162 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 05:04:02.968004 14663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:02.967701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.3:51915)
0925 05:04:02.967831 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:02.967907 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 05:04:02.968226 14661 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 05:04:02.971683 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 14533
I20260925 05:04:03.093892 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:03.093061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.1:60329)
0925 05:04:03.093559 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:03.093676 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":260,"threads_started":1}
I20260925 05:04:03.526841 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:03.526130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.2:58197)
0925 05:04:03.526536 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:03.526626 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":199,"threads_started":1}
I20260925 05:04:03.969506 14667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:03.968776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.3:40911)
0925 05:04:03.969209 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:03.969296 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":211,"threads_started":1}
I20260925 05:04:04.095436 14668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:04.094720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.1:35409)
0925 05:04:04.095183 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:04.095266 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":239,"threads_started":1}
I20260925 05:04:04.528339 14669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:04.527636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.2:36791)
0925 05:04:04.528046 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:04.528125 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":209,"threads_started":1}
I20260925 05:04:04.970995 14670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:04.970323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.3:57867)
0925 05:04:04.970766 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:04.970843 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":237,"threads_started":1}
I20260925 05:04:05.096983 14671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:05.096266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.1:51903)
0925 05:04:05.096744 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:05.096820 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":257,"threads_started":1}
I20260925 05:04:05.529903 14672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:05.529189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.2:50035)
0925 05:04:05.529604 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:05.529728 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":224,"threads_started":1}
I20260925 05:04:05.972661 14673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:05.971852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.3:41137)
0925 05:04:05.972341 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:05.972427 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":282,"threads_started":1}
I20260925 05:04:06.098433 14674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:06.097775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.1:35187)
0925 05:04:06.098176 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:06.098253 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":210,"threads_started":1}
I20260925 05:04:06.531613 14675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:06.530732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.2:44385)
0925 05:04:06.531267 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:06.531392 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":263,"threads_started":1}
I20260925 05:04:06.974186 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:06.973487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.3:35631)
0925 05:04:06.973928 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:06.974004 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":214,"threads_started":1}
I20260925 05:04:07.099957 14677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:07.099243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.1:47501)
0925 05:04:07.099723 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:07.099800 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":258,"threads_started":1}
I20260925 05:04:07.533226 14678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:07.532480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.2:33335)
0925 05:04:07.532987 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:07.533062 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":299,"threads_started":1}
I20260925 05:04:07.975689 14679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:07.975009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.3:53077)
0925 05:04:07.975451 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:07.975528 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":227,"threads_started":1}
I20260925 05:04:08.101564 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:08.100861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.1:51335)
0925 05:04:08.101302 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:08.101391 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":222,"threads_started":1}
I20260925 05:04:08.534910 14681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:08.534052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.2:40385)
0925 05:04:08.534605 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:08.534699 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":322,"threads_started":1}
I20260925 05:04:08.977250 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:08.976607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.3:42569)
0925 05:04:08.977013 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:08.977101 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":201,"threads_started":1}
I20260925 05:04:09.103404 14683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:09.102504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.1:51915)
0925 05:04:09.103111 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:09.103197 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":287,"threads_started":1}
I20260925 05:04:09.536742 14684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:09.535879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.2:52637)
0925 05:04:09.536341 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:09.536421 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":202,"threads_started":1}
I20260925 05:04:09.978995 14685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:09.978220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.3:45825)
0925 05:04:09.978701 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:09.978779 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":214,"threads_started":1}
I20260925 05:04:10.104996 14686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:10.104270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.1:40355)
0925 05:04:10.104754 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:10.104838 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":215,"threads_started":1}
I20260925 05:04:10.538326 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:10.537593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.2:32939)
0925 05:04:10.538085 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:10.538176 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":233,"threads_started":1}
I20260925 05:04:10.980474 14688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:10.979853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.3:57669)
0925 05:04:10.980254 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:10.980330 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":203,"threads_started":1}
I20260925 05:04:11.106542 14689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:11.105860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.1:49493)
0925 05:04:11.106304 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:11.106380 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":221,"threads_started":1}
I20260925 05:04:11.544780 14690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:11.542217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.2:41909)
0925 05:04:11.544375 (+ 2158us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:11.544487 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2004,"thread_start_us":1667,"threads_started":1}
I20260925 05:04:11.982239 14691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:11.981380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.3:46485)
0925 05:04:11.981884 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:11.981994 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":231,"threads_started":1}
I20260925 05:04:12.107981 14692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:12.107341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.1:55157)
0925 05:04:12.107753 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:12.107836 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":214,"threads_started":1}
I20260925 05:04:12.546275 14693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:12.545658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.2:50983)
0925 05:04:12.546048 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:12.546122 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":212,"threads_started":1}
I20260925 05:04:12.983686 14694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:12.983067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.3:47153)
0925 05:04:12.983470 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:12.983543 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":228,"threads_started":1}
I20260925 05:04:13.109632 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:13.108918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.1:54643)
0925 05:04:13.109360 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:13.109451 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":214,"threads_started":1}
I20260925 05:04:13.547783 14696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:13.547086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.2:42141)
0925 05:04:13.547517 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:13.547609 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":214,"threads_started":1}
I20260925 05:04:13.985294 14697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:13.984615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.3:34377)
0925 05:04:13.985029 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:13.985129 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":214,"threads_started":1}
I20260925 05:04:14.111352 14698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:14.110576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.1:41085)
0925 05:04:14.111109 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:14.111200 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":275,"threads_started":1}
I20260925 05:04:14.549408 14699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:14.548714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.2:38313)
0925 05:04:14.549187 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:14.549260 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":253,"threads_started":1}
I20260925 05:04:14.986860 14700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:14.986176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.3:50265)
0925 05:04:14.986610 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:14.986696 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":224,"threads_started":1}
I20260925 05:04:15.113127 14701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:15.112352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.1:59739)
0925 05:04:15.112845 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:15.112950 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":264,"threads_started":1}
I20260925 05:04:15.551153 14702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:15.550342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.2:44441)
0925 05:04:15.550869 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:15.550958 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":269,"threads_started":1}
I20260925 05:04:15.988411 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:15.987696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.3:35543)
0925 05:04:15.988146 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:15.988235 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":224,"threads_started":1}
I20260925 05:04:16.114640 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:16.113983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.1:43603)
0925 05:04:16.114390 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:16.114467 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":219,"threads_started":1}
I20260925 05:04:16.552838 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:16.552103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.2:34015)
0925 05:04:16.552515 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:16.552658 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":212,"threads_started":1}
I20260925 05:04:16.989948 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:16.989279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.3:53189)
0925 05:04:16.989684 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:16.989780 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":208,"threads_started":1}
I20260925 05:04:17.116446 14707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:17.115724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.1:60143)
0925 05:04:17.116172 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:17.116257 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":237,"threads_started":1}
I20260925 05:04:17.554466 14708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:17.553799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.2:40423)
0925 05:04:17.554185 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:17.554295 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":201,"threads_started":1}
I20260925 05:04:17.973631 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 14237
I20260925 05:04:17.991796 14710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:17.991030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40301 (local address 127.30.247.3:58635)
0925 05:04:17.991526 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:17.991628 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:40301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":283,"threads_started":1}
I20260925 05:04:17.996989 14709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:17.974723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49454 (local address 127.30.247.1:45219)
0925 05:04:17.975383 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:17.975402 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:17.975445 (+ 43us) server_negotiation.cc:407] Connection header received
0925 05:04:17.977768 (+ 2323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:17.977777 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:17.978207 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:17.978503 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:04:17.979935 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:17.981269 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:17.982583 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:17.983266 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:17.983411 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:17.990849 (+ 7438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:04:17.990857 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:04:17.990873 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:04:17.990910 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:04:17.993900 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:17.994556 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:17.994562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:17.994568 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:17.994760 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:17.995119 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:17.995123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:17.995126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:17.995646 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:04:17.995897 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:17.996302 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 05:04:17.996713 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":321,"threads_started":1}
I20260925 05:04:17.998373 14394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:04:18.082149 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 14237
I20260925 05:04:18.098903 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 14402
I20260925 05:04:18.116876 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:18.099880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51614 (local address 127.30.247.2:35113)
0925 05:04:18.100425 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:18.100438 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:18.100474 (+ 36us) server_negotiation.cc:407] Connection header received
0925 05:04:18.102117 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:18.102126 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:18.102541 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:18.102919 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:04:18.104381 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:18.105771 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:18.106998 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:18.107654 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:18.107869 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:18.110350 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:04:18.110358 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:04:18.110373 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:04:18.110413 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:04:18.113544 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:18.114321 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:18.114330 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:18.114338 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:18.114511 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:18.114902 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:18.114907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:18.114910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:18.115391 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:04:18.115719 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:18.116127 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 05:04:18.116527 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":254,"threads_started":1}
I20260925 05:04:18.118353 14525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:04:18.200917 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 14402
I20260925 05:04:18.218595 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 14533
I20260925 05:04:18.237139 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:18.219731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51302 (local address 127.30.247.3:41937)
0925 05:04:18.220398 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:18.220413 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:18.220455 (+ 42us) server_negotiation.cc:407] Connection header received
0925 05:04:18.222198 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:18.222207 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:18.222636 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:18.222960 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:04:18.224216 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:18.225521 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:18.226821 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:18.227466 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:18.227607 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:18.230219 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:04:18.230226 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:04:18.230242 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:04:18.230276 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:04:18.233727 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:18.234488 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:18.234495 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:18.234501 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:18.234662 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:18.235054 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:18.235060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:18.235063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:18.235567 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:04:18.235888 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:18.236305 (+ 417us) server_negotiation.cc:299] Negotiation successful
0925 05:04:18.236853 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":343,"threads_started":1}
I20260925 05:04:18.238497 14656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:04:18.323096 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 14533
2026-09-25T05:04:18Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24335 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:04:18 dist-test-slave-5n86 krb5kdc[14718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:04:18 dist-test-slave-5n86 krb5kdc[14718](info): set up 2 sockets
Sep 25 05:04:18 dist-test-slave-5n86 krb5kdc[14718](info): commencing operation
krb5kdc: starting...
W20260925 05:04:19.417310 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:04:19 dist-test-slave-5n86 krb5kdc[14718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:04:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:04:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:04:19.631752 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:36613
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:36613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:04:19.948933 14731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:04:19.949260 14731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:04:19.949411 14731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:04:19.956400 14731 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 05:04:19.956522 14731 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 05:04:19.956631 14731 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 05:04:19.956728 14731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:04:19.956826 14731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:04:19.956946 14731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:04:19.957032 14731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:04:19.957108 14731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:04:19.969254 14731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:36613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:36613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:04:19.970371 14731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:04:19.971594 14731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:04:19.983390 14736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:04:19.983361 14739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:04:19.983443 14737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:04:19.984071 14731 server_base.cc:1034] running on GCE node
I20260925 05:04:19.984881 14731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:04:19.986158 14731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:04:19.987398 14731 hybrid_clock.cc:648] HybridClock initialized: now 1790312659987341 us; error 38 us; skew 500 ppm
I20260925 05:04:19.987715 14731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:04:19 dist-test-slave-5n86 krb5kdc[14718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312659, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:04:19.990839 14731 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:04:19.992142 14731 webserver.cc:460] Webserver started at http://127.30.247.62:33827/ using document root <none> and password file <none>
I20260925 05:04:19.992827 14731 fs_manager.cc:359] Metadata directory not provided
I20260925 05:04:19.992910 14731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:04:19.993150 14731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:04:19.995846 14731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "f67d8ca88d4746afb89eeb62452ee1b3"
format_stamp: "Formatted at 2026-09-25 05:04:19 on dist-test-slave-5n86"
server_key: "6ebfe494484519838944b998fdb231bc"
server_key_iv: "cfe27d07e3a4248e42fe2e148cd65c0e"
server_key_version: "encryptionkey@0"
I20260925 05:04:19.996526 14731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "f67d8ca88d4746afb89eeb62452ee1b3"
format_stamp: "Formatted at 2026-09-25 05:04:19 on dist-test-slave-5n86"
server_key: "6ebfe494484519838944b998fdb231bc"
server_key_iv: "cfe27d07e3a4248e42fe2e148cd65c0e"
server_key_version: "encryptionkey@0"
I20260925 05:04:20.001433 14731 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260925 05:04:20.004631 14746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:20.005865 14731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:04:20.005986 14731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "f67d8ca88d4746afb89eeb62452ee1b3"
format_stamp: "Formatted at 2026-09-25 05:04:19 on dist-test-slave-5n86"
server_key: "6ebfe494484519838944b998fdb231bc"
server_key_iv: "cfe27d07e3a4248e42fe2e148cd65c0e"
server_key_version: "encryptionkey@0"
I20260925 05:04:20.006145 14731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:04:20.022727 14731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:04:20.026371 14731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:04:20.026568 14731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:04:20.044416 14731 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:36613
I20260925 05:04:20.044426 14797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:36613 every 8 connection(s)
I20260925 05:04:20.045677 14731 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:04:20.049151 14798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:04:20.049744 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 14731
I20260925 05:04:20.049954 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:04:20.050832 31708 external_mini_cluster.cc:1428] Setting key 4495cebe626f33a9a36e93b2d7981b96
I20260925 05:04:20.063297 14798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3: Bootstrap starting.
Sep 25 05:04:20 dist-test-slave-5n86 krb5kdc[14718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:04:20.066105 14798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3: Neither blocks nor log segments found. Creating new log.
I20260925 05:04:20.067230 14798 log.cc:824] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3: Log is configured to *not* fsync() on all Append() calls
I20260925 05:04:20.070459 14798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3: No bootstrap required, opened a new log
I20260925 05:04:20.074970 14801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:20.052821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44894 (local address 127.30.247.62:36613)
0925 05:04:20.053593 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:20.053613 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:20.053657 (+ 44us) server_negotiation.cc:407] Connection header received
0925 05:04:20.056155 (+ 2498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:20.056182 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:20.056674 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:20.057105 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:04:20.058328 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:20.060329 (+ 2001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:20.061828 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:20.062796 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:20.062994 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:20.066868 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:04:20.066879 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:04:20.066901 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:04:20.066945 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:04:20.070652 (+ 3707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:20.071315 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:20.071323 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:20.071330 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:20.071540 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:20.071902 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:20.071907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:20.071910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:20.072590 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:04:20.072905 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:20.073306 (+ 401us) server_negotiation.cc:299] Negotiation successful
0925 05:04:20.073716 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":366,"threads_started":1}
I20260925 05:04:20.079135 14798 raft_consensus.cc:348] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d8ca88d4746afb89eeb62452ee1b3" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36613 } }
I20260925 05:04:20.079433 14798 raft_consensus.cc:374] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:04:20.079504 14798 raft_consensus.cc:729] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f67d8ca88d4746afb89eeb62452ee1b3, State: Initialized, Role: FOLLOWER
I20260925 05:04:20.080010 14798 consensus_queue.cc:260] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d8ca88d4746afb89eeb62452ee1b3" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36613 } }
I20260925 05:04:20.080205 14798 raft_consensus.cc:388] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:04:20.080298 14798 raft_consensus.cc:482] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:04:20.080400 14798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:04:20.082561 14798 raft_consensus.cc:504] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d8ca88d4746afb89eeb62452ee1b3" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36613 } }
I20260925 05:04:20.082962 14798 leader_election.cc:302] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f67d8ca88d4746afb89eeb62452ee1b3; no voters:
I20260925 05:04:20.083439 14798 leader_election.cc:288] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:04:20.083564 14803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:04:20.084208 14803 raft_consensus.cc:686] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [term 1 LEADER]: Becoming Leader. State: Replica: f67d8ca88d4746afb89eeb62452ee1b3, State: Running, Role: LEADER
I20260925 05:04:20.084790 14803 consensus_queue.cc:237] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d8ca88d4746afb89eeb62452ee1b3" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36613 } }
I20260925 05:04:20.085439 14798 sys_catalog.cc:564] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:04:20.087728 14805 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f67d8ca88d4746afb89eeb62452ee1b3. Latest consensus state: current_term: 1 leader_uuid: "f67d8ca88d4746afb89eeb62452ee1b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d8ca88d4746afb89eeb62452ee1b3" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36613 } } }
I20260925 05:04:20.088450 14805 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [sys.catalog]: This master's current role is: LEADER
I20260925 05:04:20.088303 14804 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f67d8ca88d4746afb89eeb62452ee1b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d8ca88d4746afb89eeb62452ee1b3" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36613 } } }
I20260925 05:04:20.088712 14804 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [sys.catalog]: This master's current role is: LEADER
I20260925 05:04:20.089850 14812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:04:20.095058 14812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:04:20.104738 14812 catalog_manager.cc:1347] Generated new cluster ID: b082fe482e694e25915b0a16eb40d143
I20260925 05:04:20.104887 14812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:04:20.127887 14812 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:04:20.129480 14812 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:04:20.138407 14812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3: Generated new TSK 0
I20260925 05:04:20.139199 14812 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
I20260925 05:04:20.203629 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:0
--local_ip_for_outbound_sockets=127.30.247.1
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:36613
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:04:20.541487 14824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:04:20.541740 14824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:04:20.541911 14824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:04:20.548883 14824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:04:20.549002 14824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:04:20.549185 14824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:04:20.558349 14824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:36613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:04:20.559473 14824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:04:20.560700 14824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:04:20.573417 14830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:04:20.573480 14829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:04:20.573457 14832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:04:20.574187 14824 server_base.cc:1034] running on GCE node
I20260925 05:04:20.574793 14824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:04:20.575910 14824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:04:20.577119 14824 hybrid_clock.cc:648] HybridClock initialized: now 1790312660577083 us; error 33 us; skew 500 ppm
I20260925 05:04:20.577461 14824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:04:20 dist-test-slave-5n86 krb5kdc[14718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312660, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:04:20.580634 14824 init.cc:377] Logged in from keytab as kudu/127.30.247.1@KRBTEST.COM (short username kudu)
I20260925 05:04:20.581919 14824 webserver.cc:460] Webserver started at http://127.30.247.1:41357/ using document root <none> and password file <none>
I20260925 05:04:20.582551 14824 fs_manager.cc:359] Metadata directory not provided
I20260925 05:04:20.582647 14824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:04:20.583004 14824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:04:20.585949 14824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/data/instance:
uuid: "2efd1e14d8494b48b714b28b1476fb52"
format_stamp: "Formatted at 2026-09-25 05:04:20 on dist-test-slave-5n86"
server_key: "d13c5f23da204ecd2873a9dab89c50e0"
server_key_iv: "aee8b583550b09d80e2b38b3585601fa"
server_key_version: "encryptionkey@0"
I20260925 05:04:20.586635 14824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance:
uuid: "2efd1e14d8494b48b714b28b1476fb52"
format_stamp: "Formatted at 2026-09-25 05:04:20 on dist-test-slave-5n86"
server_key: "d13c5f23da204ecd2873a9dab89c50e0"
server_key_iv: "aee8b583550b09d80e2b38b3585601fa"
server_key_version: "encryptionkey@0"
I20260925 05:04:20.591938 14824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 05:04:20.595206 14839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:20.596481 14824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:04:20.596699 14824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "2efd1e14d8494b48b714b28b1476fb52"
format_stamp: "Formatted at 2026-09-25 05:04:20 on dist-test-slave-5n86"
server_key: "d13c5f23da204ecd2873a9dab89c50e0"
server_key_iv: "aee8b583550b09d80e2b38b3585601fa"
server_key_version: "encryptionkey@0"
I20260925 05:04:20.596866 14824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:04:20.613252 14824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:04:20.617261 14824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:04:20.617532 14824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:04:20.618389 14824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:04:20.619611 14824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:04:20.619680 14824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:20.619858 14824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:04:20.619910 14824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:20.651358 14824 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:36567
I20260925 05:04:20.651400 14951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:36567 every 8 connection(s)
I20260925 05:04:20.652601 14824 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:04:20.656738 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 14824
I20260925 05:04:20.656939 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance
I20260925 05:04:20.657567 31708 external_mini_cluster.cc:1428] Setting key fb167509f00a64e7025983f092b67aca
Sep 25 05:04:20 dist-test-slave-5n86 krb5kdc[14718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312660, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:04:20.672168 14954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:20.654981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:51921 (local address 127.30.247.62:36613)
0925 05:04:20.655401 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:20.655405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:20.656267 (+ 862us) server_negotiation.cc:407] Connection header received
0925 05:04:20.658292 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:20.658297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:20.658387 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:20.658503 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:04:20.659902 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:20.660655 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:20.662130 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:20.662531 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:20.662660 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:20.666030 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:04:20.666036 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:04:20.666039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:04:20.666070 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:04:20.669000 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:20.669876 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:20.669880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:20.669883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:20.670003 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:20.670484 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:20.670489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:20.670491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:20.670793 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:04:20.670913 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:20.671752 (+ 839us) server_negotiation.cc:299] Negotiation successful
0925 05:04:20.671911 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":225,"threads_started":1}
I20260925 05:04:20.673585 14955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:20.655388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.1:51921)
0925 05:04:20.656090 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:20.656130 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 05:04:20.658038 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:04:20.658727 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:04:20.658735 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:04:20.659203 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:04:20.659730 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:20.659745 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:20.660814 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:20.660820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:20.661987 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:20.661998 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:20.662756 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:20.662763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:20.662988 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:20.663649 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:04:20.663675 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:04:20.665808 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:04:20.669134 (+ 3326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:04:20.669146 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:04:20.669165 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:04:20.669753 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:04:20.670114 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:04:20.670122 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:04:20.670128 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:04:20.670353 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:04:20.670926 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:04:20.670936 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:04:20.671342 (+ 406us) client_negotiation.cc:769] Sending connection context
0925 05:04:20.671654 (+ 312us) client_negotiation.cc:240] Negotiation successful
0925 05:04:20.672199 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":281,"threads_started":1}
I20260925 05:04:20.675051 14952 heartbeater.cc:347] Connected to a master server at 127.30.247.62:36613
I20260925 05:04:20.675333 14952 heartbeater.cc:464] Registering TS with master...
I20260925 05:04:20.676112 14952 heartbeater.cc:511] Master 127.30.247.62:36613 requested a full tablet report, sending...
I20260925 05:04:20.678126 14763 ts_manager.cc:194] Registered new tserver with Master: 2efd1e14d8494b48b714b28b1476fb52 (127.30.247.1:36567)
I20260925 05:04:20.679517 14763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.1@KRBTEST.COM'} at 127.30.247.1:51921
WARNING: no policy specified for kudu/127.30.247.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
I20260925 05:04:20.711381 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:0
--local_ip_for_outbound_sockets=127.30.247.2
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:36613
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:04:21.021827 14958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:04:21.022083 14958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:04:21.022172 14958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:04:21.028889 14958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:04:21.028976 14958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:04:21.029078 14958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:04:21.037976 14958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:36613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:04:21.039002 14958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:04:21.040134 14958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:04:21.053227 14963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:04:21.053207 14964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:04:21.053544 14966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:04:21.053875 14958 server_base.cc:1034] running on GCE node
I20260925 05:04:21.054715 14958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:04:21.055603 14958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:04:21.056828 14958 hybrid_clock.cc:648] HybridClock initialized: now 1790312661056780 us; error 41 us; skew 500 ppm
I20260925 05:04:21.057123 14958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:04:21 dist-test-slave-5n86 krb5kdc[14718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312661, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:04:21.060247 14958 init.cc:377] Logged in from keytab as kudu/127.30.247.2@KRBTEST.COM (short username kudu)
I20260925 05:04:21.061713 14958 webserver.cc:460] Webserver started at http://127.30.247.2:32893/ using document root <none> and password file <none>
I20260925 05:04:21.062363 14958 fs_manager.cc:359] Metadata directory not provided
I20260925 05:04:21.062505 14958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:04:21.062796 14958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:04:21.065551 14958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/data/instance:
uuid: "46c2a23747a5451ab3e440a2b7549491"
format_stamp: "Formatted at 2026-09-25 05:04:21 on dist-test-slave-5n86"
server_key: "c6ac0cf3e4ac96cb8852b7eccd2e9ba9"
server_key_iv: "e334f34cc78c0d54a3e477c2ec9bf809"
server_key_version: "encryptionkey@0"
I20260925 05:04:21.066251 14958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance:
uuid: "46c2a23747a5451ab3e440a2b7549491"
format_stamp: "Formatted at 2026-09-25 05:04:21 on dist-test-slave-5n86"
server_key: "c6ac0cf3e4ac96cb8852b7eccd2e9ba9"
server_key_iv: "e334f34cc78c0d54a3e477c2ec9bf809"
server_key_version: "encryptionkey@0"
I20260925 05:04:21.071007 14958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 05:04:21.074327 14973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:21.075513 14958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:04:21.075681 14958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "46c2a23747a5451ab3e440a2b7549491"
format_stamp: "Formatted at 2026-09-25 05:04:21 on dist-test-slave-5n86"
server_key: "c6ac0cf3e4ac96cb8852b7eccd2e9ba9"
server_key_iv: "e334f34cc78c0d54a3e477c2ec9bf809"
server_key_version: "encryptionkey@0"
I20260925 05:04:21.075865 14958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:04:21.093760 14958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:04:21.097977 14958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:04:21.098263 14958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:04:21.099217 14958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:04:21.100713 14958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:04:21.100796 14958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:21.100935 14958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:04:21.100996 14958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:21.132406 14958 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:46345
I20260925 05:04:21.132452 15085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:46345 every 8 connection(s)
I20260925 05:04:21.133709 14958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:04:21.140200 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 14958
I20260925 05:04:21.140395 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance
I20260925 05:04:21.140988 31708 external_mini_cluster.cc:1428] Setting key ec8626d9ce86bce1a2789dc6e704b183
Sep 25 05:04:21 dist-test-slave-5n86 krb5kdc[14718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312661, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:04:21.155042 14954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:21.136323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:54893 (local address 127.30.247.62:36613)
0925 05:04:21.136504 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:21.136509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:21.137661 (+ 1152us) server_negotiation.cc:407] Connection header received
0925 05:04:21.139413 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:21.139417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:21.139488 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:21.139593 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:04:21.141650 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.142459 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.144677 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.145115 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.145219 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:21.149305 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:04:21.149310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:04:21.149313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:04:21.149346 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:04:21.151901 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:21.152635 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:21.152642 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:21.152650 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:21.152774 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:21.153188 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:21.153193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:21.153196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:21.153452 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:04:21.153605 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:21.154331 (+ 726us) server_negotiation.cc:299] Negotiation successful
0925 05:04:21.154594 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 05:04:21.155836 15088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:21.136783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.2:54893)
0925 05:04:21.137486 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:21.137529 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 05:04:21.139151 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:04:21.139816 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:04:21.139827 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:04:21.140620 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:04:21.141407 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:21.141432 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.142631 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.142636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:21.144494 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:21.144510 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.145347 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.145355 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:21.145672 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:21.146397 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:04:21.146436 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:04:21.149075 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:04:21.152044 (+ 2969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:04:21.152053 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:04:21.152069 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:04:21.152476 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:04:21.152887 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:04:21.152892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:04:21.152896 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:04:21.153071 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:04:21.153579 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:04:21.153586 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:04:21.153914 (+ 328us) client_negotiation.cc:769] Sending connection context
0925 05:04:21.154193 (+ 279us) client_negotiation.cc:240] Negotiation successful
0925 05:04:21.154736 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":268,"threads_started":1}
I20260925 05:04:21.157330 15086 heartbeater.cc:347] Connected to a master server at 127.30.247.62:36613
I20260925 05:04:21.157652 15086 heartbeater.cc:464] Registering TS with master...
I20260925 05:04:21.158502 15086 heartbeater.cc:511] Master 127.30.247.62:36613 requested a full tablet report, sending...
I20260925 05:04:21.160038 14763 ts_manager.cc:194] Registered new tserver with Master: 46c2a23747a5451ab3e440a2b7549491 (127.30.247.2:46345)
I20260925 05:04:21.161303 14763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.2@KRBTEST.COM'} at 127.30.247.2:54893
WARNING: no policy specified for kudu/127.30.247.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
I20260925 05:04:21.195673 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:0
--local_ip_for_outbound_sockets=127.30.247.3
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:36613
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:04:21.504936 15091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:04:21.505183 15091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:04:21.505273 15091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:04:21.512326 15091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:04:21.512419 15091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:04:21.512575 15091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:04:21.521704 15091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:36613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:04:21.522759 15091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:04:21.523936 15091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:04:21.536823 15097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:04:21.536881 15096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:04:21.536906 15099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:04:21.537626 15091 server_base.cc:1034] running on GCE node
I20260925 05:04:21.538116 15091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:04:21.538905 15091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:04:21.540124 15091 hybrid_clock.cc:648] HybridClock initialized: now 1790312661540084 us; error 50 us; skew 500 ppm
I20260925 05:04:21.540423 15091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:04:21 dist-test-slave-5n86 krb5kdc[14718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312661, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:04:21.543516 15091 init.cc:377] Logged in from keytab as kudu/127.30.247.3@KRBTEST.COM (short username kudu)
I20260925 05:04:21.544940 15091 webserver.cc:460] Webserver started at http://127.30.247.3:34513/ using document root <none> and password file <none>
I20260925 05:04:21.545574 15091 fs_manager.cc:359] Metadata directory not provided
I20260925 05:04:21.545717 15091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:04:21.546002 15091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:04:21.548725 15091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/data/instance:
uuid: "501237da7c2c4854b627f5015f66fb70"
format_stamp: "Formatted at 2026-09-25 05:04:21 on dist-test-slave-5n86"
server_key: "e19eb6e09693392c367bc2b032278067"
server_key_iv: "4470128858c23210cbf297db32f12ee7"
server_key_version: "encryptionkey@0"
I20260925 05:04:21.549464 15091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance:
uuid: "501237da7c2c4854b627f5015f66fb70"
format_stamp: "Formatted at 2026-09-25 05:04:21 on dist-test-slave-5n86"
server_key: "e19eb6e09693392c367bc2b032278067"
server_key_iv: "4470128858c23210cbf297db32f12ee7"
server_key_version: "encryptionkey@0"
I20260925 05:04:21.554301 15091 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 05:04:21.557653 15106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:21.558892 15091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:04:21.559062 15091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "501237da7c2c4854b627f5015f66fb70"
format_stamp: "Formatted at 2026-09-25 05:04:21 on dist-test-slave-5n86"
server_key: "e19eb6e09693392c367bc2b032278067"
server_key_iv: "4470128858c23210cbf297db32f12ee7"
server_key_version: "encryptionkey@0"
I20260925 05:04:21.559259 15091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:04:21.571702 15091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:04:21.575599 15091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:04:21.575874 15091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:04:21.576802 15091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:04:21.578075 15091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:04:21.578174 15091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:21.578313 15091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:04:21.578390 15091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:21.611322 15091 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:35489
I20260925 05:04:21.611403 15218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:35489 every 8 connection(s)
I20260925 05:04:21.612564 15091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:04:21.614566 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 15091
I20260925 05:04:21.614730 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance
I20260925 05:04:21.615222 31708 external_mini_cluster.cc:1428] Setting key cbb49ccabcb913061c51e89a180daa4d
Sep 25 05:04:21 dist-test-slave-5n86 krb5kdc[14718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312661, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:04:21.631488 14954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:21.615198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:44107 (local address 127.30.247.62:36613)
0925 05:04:21.615423 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:21.615430 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:21.616766 (+ 1336us) server_negotiation.cc:407] Connection header received
0925 05:04:21.618511 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:21.618515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:21.618603 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:21.618684 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:04:21.620027 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.620820 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.622245 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.622640 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.622729 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:21.625957 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:04:21.625962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:04:21.625965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:04:21.625995 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:04:21.628603 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:21.629390 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:21.629396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:21.629400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:21.629525 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:21.629976 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:21.629980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:21.629982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:21.630329 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:04:21.630443 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:21.631077 (+ 634us) server_negotiation.cc:299] Negotiation successful
0925 05:04:21.631233 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 05:04:21.632288 15221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:21.615710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.3:44107)
0925 05:04:21.616600 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:21.616651 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 05:04:21.618265 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:04:21.618878 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:04:21.618886 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:04:21.619360 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:04:21.619894 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:21.619911 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.620946 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.620949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:21.622125 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:21.622136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.622810 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.622814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:21.623000 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:21.623577 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:04:21.623597 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:04:21.625763 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:04:21.628742 (+ 2979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:04:21.628750 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:04:21.628763 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:04:21.629268 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:04:21.629632 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:04:21.629637 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:04:21.629641 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:04:21.629867 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:04:21.630451 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:04:21.630457 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:04:21.630750 (+ 293us) client_negotiation.cc:769] Sending connection context
0925 05:04:21.630966 (+ 216us) client_negotiation.cc:240] Negotiation successful
0925 05:04:21.631296 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":375,"threads_started":1}
I20260925 05:04:21.633591 15219 heartbeater.cc:347] Connected to a master server at 127.30.247.62:36613
I20260925 05:04:21.633889 15219 heartbeater.cc:464] Registering TS with master...
I20260925 05:04:21.634546 15219 heartbeater.cc:511] Master 127.30.247.62:36613 requested a full tablet report, sending...
I20260925 05:04:21.635913 14763 ts_manager.cc:194] Registered new tserver with Master: 501237da7c2c4854b627f5015f66fb70 (127.30.247.3:35489)
I20260925 05:04:21.636731 14763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.3@KRBTEST.COM'} at 127.30.247.3:44107
I20260925 05:04:21.642668 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:04:21.660007 14954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:21.647090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44898 (local address 127.30.247.62:36613)
0925 05:04:21.647334 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:21.647343 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:21.647654 (+ 311us) server_negotiation.cc:407] Connection header received
0925 05:04:21.647786 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:21.647791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:21.647852 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:21.647933 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:04:21.649570 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.650262 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.651835 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.652224 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.652375 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:21.654862 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:04:21.654868 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:04:21.654871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:04:21.654902 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:04:21.657608 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:21.658216 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:21.658220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:21.658222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:21.658348 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:21.658648 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:21.658650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:21.658652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:21.658952 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:04:21.659125 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:21.659618 (+ 493us) server_negotiation.cc:299] Negotiation successful
0925 05:04:21.659766 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 05:04:21.664389 14763 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:04:21.671175 14763 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:04:21.682258 14952 heartbeater.cc:503] Master 127.30.247.62:36613 was elected leader, sending a full tablet report...
I20260925 05:04:21.700841 15231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:21.682779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:35489 (local address 127.0.0.1:43774)
0925 05:04:21.685275 (+ 2496us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:21.685311 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 05:04:21.685495 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:04:21.687913 (+ 2418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:04:21.687924 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:04:21.687971 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:04:21.688469 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:21.688485 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.692389 (+ 3904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.692397 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:21.696454 (+ 4057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:21.696473 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.698878 (+ 2405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.698887 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:21.699163 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:21.699251 (+ 88us) client_negotiation.cc:769] Sending connection context
0925 05:04:21.699370 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 05:04:21.700433 (+ 1063us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2261,"thread_start_us":2112,"threads_started":1}
I20260925 05:04:21.702087 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:21.682833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43774 (local address 127.30.247.3:35489)
0925 05:04:21.686102 (+ 3269us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:21.686110 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:21.686146 (+ 36us) server_negotiation.cc:407] Connection header received
0925 05:04:21.686259 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:21.686266 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:21.686572 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:21.688722 (+ 2150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:04:21.689055 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.692215 (+ 3160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.696721 (+ 4506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.698609 (+ 1888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.701569 (+ 2960us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:21.701724 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:21.701776 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 05:04:21.701880 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3081,"thread_start_us":2929,"threads_started":1}
I20260925 05:04:21.706092 15154 tablet_service.cc:1467] Processing CreateTablet for tablet b387151ce9a54e92b15d2d450b8d9bbb (DEFAULT_TABLE table=test-table [id=9ec86c42359047898994a32663057b90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:04:21.706936 15235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:21.682876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:46345 (local address 127.0.0.1:50156)
0925 05:04:21.694493 (+ 11617us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:21.694517 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 05:04:21.694657 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:04:21.696942 (+ 2285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:04:21.696949 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:04:21.696985 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:04:21.697475 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:21.697489 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.699966 (+ 2477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.699974 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:21.703756 (+ 3782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:21.703769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.706073 (+ 2304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.706084 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:21.706322 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:21.706355 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 05:04:21.706436 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 05:04:21.706547 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11449,"mutex_wait_us":29,"thread_start_us":20180,"threads_started":1}
I20260925 05:04:21.707645 15154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b387151ce9a54e92b15d2d450b8d9bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:04:21.707744 15234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:21.683325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50156 (local address 127.30.247.2:46345)
0925 05:04:21.695796 (+ 12471us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:21.695808 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:21.695842 (+ 34us) server_negotiation.cc:407] Connection header received
0925 05:04:21.695967 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:21.695978 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:21.696252 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:21.696468 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:04:21.697725 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.699746 (+ 2021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.703920 (+ 4174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.705080 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.705236 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:21.705420 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:21.707277 (+ 1857us) server_negotiation.cc:299] Negotiation successful
0925 05:04:21.707428 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12269,"thread_start_us":13369,"threads_started":1}
I20260925 05:04:21.708634 15231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:21.685856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:36567 (local address 127.0.0.1:56466)
0925 05:04:21.701070 (+ 15214us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:21.701085 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:04:21.701183 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:04:21.702701 (+ 1518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:04:21.702705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:04:21.702739 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:04:21.702897 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:21.702909 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.704455 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.704460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:21.706299 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:21.706313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.707693 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.707698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:21.708027 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:21.708068 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 05:04:21.708163 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 05:04:21.708282 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15138,"mutex_wait_us":25,"thread_start_us":8305,"threads_started":1}
I20260925 05:04:21.708715 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:21.686317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56466 (local address 127.30.247.1:36567)
0925 05:04:21.700104 (+ 13787us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:21.700112 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:21.702297 (+ 2185us) server_negotiation.cc:407] Connection header received
0925 05:04:21.702389 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:21.702394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:21.702539 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:21.702713 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:04:21.703081 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.704285 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.706449 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.707532 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.707669 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:21.707868 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:21.708223 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 05:04:21.708375 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13639,"thread_start_us":13789,"threads_started":1}
I20260925 05:04:21.710974 15021 tablet_service.cc:1467] Processing CreateTablet for tablet b387151ce9a54e92b15d2d450b8d9bbb (DEFAULT_TABLE table=test-table [id=9ec86c42359047898994a32663057b90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:04:21.711985 15021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b387151ce9a54e92b15d2d450b8d9bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:04:21.712239 14887 tablet_service.cc:1467] Processing CreateTablet for tablet b387151ce9a54e92b15d2d450b8d9bbb (DEFAULT_TABLE table=test-table [id=9ec86c42359047898994a32663057b90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:04:21.713886 14887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b387151ce9a54e92b15d2d450b8d9bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:04:21.722879 15237 tablet_bootstrap.cc:492] T b387151ce9a54e92b15d2d450b8d9bbb P 501237da7c2c4854b627f5015f66fb70: Bootstrap starting.
I20260925 05:04:21.723609 15238 tablet_bootstrap.cc:492] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491: Bootstrap starting.
I20260925 05:04:21.726788 15239 tablet_bootstrap.cc:492] T b387151ce9a54e92b15d2d450b8d9bbb P 2efd1e14d8494b48b714b28b1476fb52: Bootstrap starting.
I20260925 05:04:21.727095 15238 tablet_bootstrap.cc:654] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491: Neither blocks nor log segments found. Creating new log.
I20260925 05:04:21.727741 15237 tablet_bootstrap.cc:654] T b387151ce9a54e92b15d2d450b8d9bbb P 501237da7c2c4854b627f5015f66fb70: Neither blocks nor log segments found. Creating new log.
I20260925 05:04:21.728365 15238 log.cc:824] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491: Log is configured to *not* fsync() on all Append() calls
I20260925 05:04:21.728950 15237 log.cc:824] T b387151ce9a54e92b15d2d450b8d9bbb P 501237da7c2c4854b627f5015f66fb70: Log is configured to *not* fsync() on all Append() calls
I20260925 05:04:21.729842 15239 tablet_bootstrap.cc:654] T b387151ce9a54e92b15d2d450b8d9bbb P 2efd1e14d8494b48b714b28b1476fb52: Neither blocks nor log segments found. Creating new log.
I20260925 05:04:21.730981 15239 log.cc:824] T b387151ce9a54e92b15d2d450b8d9bbb P 2efd1e14d8494b48b714b28b1476fb52: Log is configured to *not* fsync() on all Append() calls
I20260925 05:04:21.731158 15238 tablet_bootstrap.cc:492] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491: No bootstrap required, opened a new log
I20260925 05:04:21.731429 15238 ts_tablet_manager.cc:1397] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 05:04:21.732010 15237 tablet_bootstrap.cc:492] T b387151ce9a54e92b15d2d450b8d9bbb P 501237da7c2c4854b627f5015f66fb70: No bootstrap required, opened a new log
I20260925 05:04:21.732296 15237 ts_tablet_manager.cc:1397] T b387151ce9a54e92b15d2d450b8d9bbb P 501237da7c2c4854b627f5015f66fb70: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
I20260925 05:04:21.735333 15239 tablet_bootstrap.cc:492] T b387151ce9a54e92b15d2d450b8d9bbb P 2efd1e14d8494b48b714b28b1476fb52: No bootstrap required, opened a new log
I20260925 05:04:21.735577 15239 ts_tablet_manager.cc:1397] T b387151ce9a54e92b15d2d450b8d9bbb P 2efd1e14d8494b48b714b28b1476fb52: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260925 05:04:21.738690 15238 raft_consensus.cc:348] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501237da7c2c4854b627f5015f66fb70" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35489 } } peers { permanent_uuid: "46c2a23747a5451ab3e440a2b7549491" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46345 } } peers { permanent_uuid: "2efd1e14d8494b48b714b28b1476fb52" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36567 } }
I20260925 05:04:21.739025 15238 raft_consensus.cc:374] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:04:21.739147 15238 raft_consensus.cc:729] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c2a23747a5451ab3e440a2b7549491, State: Initialized, Role: FOLLOWER
I20260925 05:04:21.739692 15238 consensus_queue.cc:260] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501237da7c2c4854b627f5015f66fb70" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35489 } } peers { permanent_uuid: "46c2a23747a5451ab3e440a2b7549491" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46345 } } peers { permanent_uuid: "2efd1e14d8494b48b714b28b1476fb52" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36567 } }
I20260925 05:04:21.740734 15086 heartbeater.cc:503] Master 127.30.247.62:36613 was elected leader, sending a full tablet report...
I20260925 05:04:21.741317 15238 ts_tablet_manager.cc:1428] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260925 05:04:21.742584 15237 raft_consensus.cc:348] T b387151ce9a54e92b15d2d450b8d9bbb P 501237da7c2c4854b627f5015f66fb70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501237da7c2c4854b627f5015f66fb70" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35489 } } peers { permanent_uuid: "46c2a23747a5451ab3e440a2b7549491" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46345 } } peers { permanent_uuid: "2efd1e14d8494b48b714b28b1476fb52" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36567 } }
I20260925 05:04:21.742986 15237 raft_consensus.cc:374] T b387151ce9a54e92b15d2d450b8d9bbb P 501237da7c2c4854b627f5015f66fb70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:04:21.743073 15237 raft_consensus.cc:729] T b387151ce9a54e92b15d2d450b8d9bbb P 501237da7c2c4854b627f5015f66fb70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501237da7c2c4854b627f5015f66fb70, State: Initialized, Role: FOLLOWER
I20260925 05:04:21.743580 15237 consensus_queue.cc:260] T b387151ce9a54e92b15d2d450b8d9bbb P 501237da7c2c4854b627f5015f66fb70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501237da7c2c4854b627f5015f66fb70" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35489 } } peers { permanent_uuid: "46c2a23747a5451ab3e440a2b7549491" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46345 } } peers { permanent_uuid: "2efd1e14d8494b48b714b28b1476fb52" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36567 } }
I20260925 05:04:21.744683 15219 heartbeater.cc:503] Master 127.30.247.62:36613 was elected leader, sending a full tablet report...
I20260925 05:04:21.745290 15237 ts_tablet_manager.cc:1428] T b387151ce9a54e92b15d2d450b8d9bbb P 501237da7c2c4854b627f5015f66fb70: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260925 05:04:21.745756 15239 raft_consensus.cc:348] T b387151ce9a54e92b15d2d450b8d9bbb P 2efd1e14d8494b48b714b28b1476fb52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501237da7c2c4854b627f5015f66fb70" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35489 } } peers { permanent_uuid: "46c2a23747a5451ab3e440a2b7549491" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46345 } } peers { permanent_uuid: "2efd1e14d8494b48b714b28b1476fb52" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36567 } }
I20260925 05:04:21.746142 15239 raft_consensus.cc:374] T b387151ce9a54e92b15d2d450b8d9bbb P 2efd1e14d8494b48b714b28b1476fb52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:04:21.746255 15239 raft_consensus.cc:729] T b387151ce9a54e92b15d2d450b8d9bbb P 2efd1e14d8494b48b714b28b1476fb52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efd1e14d8494b48b714b28b1476fb52, State: Initialized, Role: FOLLOWER
I20260925 05:04:21.746770 15239 consensus_queue.cc:260] T b387151ce9a54e92b15d2d450b8d9bbb P 2efd1e14d8494b48b714b28b1476fb52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501237da7c2c4854b627f5015f66fb70" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35489 } } peers { permanent_uuid: "46c2a23747a5451ab3e440a2b7549491" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46345 } } peers { permanent_uuid: "2efd1e14d8494b48b714b28b1476fb52" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36567 } }
I20260925 05:04:21.748528 15239 ts_tablet_manager.cc:1428] T b387151ce9a54e92b15d2d450b8d9bbb P 2efd1e14d8494b48b714b28b1476fb52: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260925 05:04:21.823894 15243 raft_consensus.cc:482] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:04:21.824222 15243 raft_consensus.cc:504] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501237da7c2c4854b627f5015f66fb70" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35489 } } peers { permanent_uuid: "46c2a23747a5451ab3e440a2b7549491" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46345 } } peers { permanent_uuid: "2efd1e14d8494b48b714b28b1476fb52" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36567 } }
I20260925 05:04:21.825867 15243 leader_election.cc:288] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 501237da7c2c4854b627f5015f66fb70 (127.30.247.3:35489), 2efd1e14d8494b48b714b28b1476fb52 (127.30.247.1:36567)
I20260925 05:04:21.832366 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:21.826332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:55899 (local address 127.30.247.3:35489)
0925 05:04:21.826515 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:21.826520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:21.826968 (+ 448us) server_negotiation.cc:407] Connection header received
0925 05:04:21.827252 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:21.827259 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:21.827338 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:21.827414 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:04:21.828061 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.828897 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.830688 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.831435 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.831615 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:21.831690 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:21.831982 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 05:04:21.832103 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 05:04:21.832567 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:21.826171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:35489 (local address 127.30.247.2:55899)
0925 05:04:21.826865 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:21.826884 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:04:21.827018 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:04:21.827580 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:04:21.827583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:04:21.827653 (+ 70us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:04:21.827911 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:21.827919 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.829031 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.829038 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:21.830536 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:21.830547 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.831644 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.831650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:21.831821 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:21.831844 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 05:04:21.831959 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 05:04:21.832084 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":28,"spinlock_wait_cycles":28928,"thread_start_us":357,"threads_started":1}
I20260925 05:04:21.832593 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:21.827110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:57671 (local address 127.30.247.1:36567)
0925 05:04:21.827332 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:21.827339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:21.827572 (+ 233us) server_negotiation.cc:407] Connection header received
0925 05:04:21.827766 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:21.827771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:21.827837 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:21.827930 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:04:21.828466 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.829202 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.830887 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.831639 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.831808 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:21.831888 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:21.832182 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 05:04:21.832313 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 05:04:21.832713 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:21.826944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:36567 (local address 127.30.247.2:57671)
0925 05:04:21.827469 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:21.827482 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:04:21.827627 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:04:21.827943 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:04:21.827947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:04:21.827969 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:04:21.828326 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:21.828335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.829339 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.829346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:21.830731 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:21.830741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.831874 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.831880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:21.832033 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:21.832055 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 05:04:21.832157 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 05:04:21.832277 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":1,"thread_start_us":264,"threads_started":1}
I20260925 05:04:21.833740 15174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b387151ce9a54e92b15d2d450b8d9bbb" candidate_uuid: "46c2a23747a5451ab3e440a2b7549491" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501237da7c2c4854b627f5015f66fb70" is_pre_election: true
I20260925 05:04:21.833742 14907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b387151ce9a54e92b15d2d450b8d9bbb" candidate_uuid: "46c2a23747a5451ab3e440a2b7549491" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efd1e14d8494b48b714b28b1476fb52" is_pre_election: true
I20260925 05:04:21.834331 15174 raft_consensus.cc:2454] T b387151ce9a54e92b15d2d450b8d9bbb P 501237da7c2c4854b627f5015f66fb70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46c2a23747a5451ab3e440a2b7549491 in term 0.
I20260925 05:04:21.834319 14907 raft_consensus.cc:2454] T b387151ce9a54e92b15d2d450b8d9bbb P 2efd1e14d8494b48b714b28b1476fb52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46c2a23747a5451ab3e440a2b7549491 in term 0.
I20260925 05:04:21.835039 14977 leader_election.cc:302] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46c2a23747a5451ab3e440a2b7549491, 501237da7c2c4854b627f5015f66fb70; no voters:
I20260925 05:04:21.835421 15243 raft_consensus.cc:2781] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:04:21.835597 15243 raft_consensus.cc:482] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:04:21.835709 15243 raft_consensus.cc:3037] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:04:21.837915 15243 raft_consensus.cc:504] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501237da7c2c4854b627f5015f66fb70" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35489 } } peers { permanent_uuid: "46c2a23747a5451ab3e440a2b7549491" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46345 } } peers { permanent_uuid: "2efd1e14d8494b48b714b28b1476fb52" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36567 } }
I20260925 05:04:21.838387 15243 leader_election.cc:288] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491 [CANDIDATE]: Term 1 election: Requested vote from peers 501237da7c2c4854b627f5015f66fb70 (127.30.247.3:35489), 2efd1e14d8494b48b714b28b1476fb52 (127.30.247.1:36567)
I20260925 05:04:21.838824 15174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b387151ce9a54e92b15d2d450b8d9bbb" candidate_uuid: "46c2a23747a5451ab3e440a2b7549491" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501237da7c2c4854b627f5015f66fb70"
I20260925 05:04:21.838904 14907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b387151ce9a54e92b15d2d450b8d9bbb" candidate_uuid: "46c2a23747a5451ab3e440a2b7549491" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efd1e14d8494b48b714b28b1476fb52"
I20260925 05:04:21.839004 15174 raft_consensus.cc:3037] T b387151ce9a54e92b15d2d450b8d9bbb P 501237da7c2c4854b627f5015f66fb70 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:04:21.839159 14907 raft_consensus.cc:3037] T b387151ce9a54e92b15d2d450b8d9bbb P 2efd1e14d8494b48b714b28b1476fb52 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:04:21.841249 15174 raft_consensus.cc:2454] T b387151ce9a54e92b15d2d450b8d9bbb P 501237da7c2c4854b627f5015f66fb70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c2a23747a5451ab3e440a2b7549491 in term 1.
I20260925 05:04:21.841414 14907 raft_consensus.cc:2454] T b387151ce9a54e92b15d2d450b8d9bbb P 2efd1e14d8494b48b714b28b1476fb52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c2a23747a5451ab3e440a2b7549491 in term 1.
I20260925 05:04:21.841648 14977 leader_election.cc:302] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46c2a23747a5451ab3e440a2b7549491, 501237da7c2c4854b627f5015f66fb70; no voters:
I20260925 05:04:21.841939 15243 raft_consensus.cc:2781] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:04:21.842386 15243 raft_consensus.cc:686] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491 [term 1 LEADER]: Becoming Leader. State: Replica: 46c2a23747a5451ab3e440a2b7549491, State: Running, Role: LEADER
I20260925 05:04:21.842844 15243 consensus_queue.cc:237] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501237da7c2c4854b627f5015f66fb70" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35489 } } peers { permanent_uuid: "46c2a23747a5451ab3e440a2b7549491" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46345 } } peers { permanent_uuid: "2efd1e14d8494b48b714b28b1476fb52" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36567 } }
I20260925 05:04:21.847584 14762 catalog_manager.cc:5495] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46c2a23747a5451ab3e440a2b7549491 (127.30.247.2). New cstate: current_term: 1 leader_uuid: "46c2a23747a5451ab3e440a2b7549491" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501237da7c2c4854b627f5015f66fb70" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46c2a23747a5451ab3e440a2b7549491" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efd1e14d8494b48b714b28b1476fb52" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36567 } health_report { overall_health: UNKNOWN } } }
W20260925 05:04:21.864182 15220 tablet.cc:2369] T b387151ce9a54e92b15d2d450b8d9bbb P 501237da7c2c4854b627f5015f66fb70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:04:21.884855 15234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:21.879410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50164 (local address 127.30.247.2:46345)
0925 05:04:21.879629 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:21.879635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:21.879655 (+ 20us) server_negotiation.cc:407] Connection header received
0925 05:04:21.879847 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:21.879851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:21.879940 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:21.880106 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:04:21.880498 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.881270 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.882726 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.883133 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.883278 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:21.883772 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:04:21.884244 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:04:21.884410 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:21.884470 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 05:04:21.884573 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 05:04:21.885749 15087 tablet.cc:2369] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:04:21.893247 14907 raft_consensus.cc:1261] T b387151ce9a54e92b15d2d450b8d9bbb P 2efd1e14d8494b48b714b28b1476fb52 [term 1 FOLLOWER]: Refusing update from remote peer 46c2a23747a5451ab3e440a2b7549491: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:04:21.893246 15174 raft_consensus.cc:1261] T b387151ce9a54e92b15d2d450b8d9bbb P 501237da7c2c4854b627f5015f66fb70 [term 1 FOLLOWER]: Refusing update from remote peer 46c2a23747a5451ab3e440a2b7549491: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:04:21.894217 15248 consensus_queue.cc:1035] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501237da7c2c4854b627f5015f66fb70" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:04:21.894578 15243 consensus_queue.cc:1035] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efd1e14d8494b48b714b28b1476fb52" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 05:04:21.905032 14953 tablet.cc:2369] T b387151ce9a54e92b15d2d450b8d9bbb P 2efd1e14d8494b48b714b28b1476fb52: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:04:21.905975 15253 mvcc.cc:205] Tried to move back new op lower bound from 7333120663104249856 to 7333120662912167936. Current Snapshot: MvccSnapshot[applied={T|T < 7333120663104249856}]
I20260925 05:04:21.906597 15251 mvcc.cc:205] Tried to move back new op lower bound from 7333120663104249856 to 7333120662912167936. Current Snapshot: MvccSnapshot[applied={T|T < 7333120663104249856}]
I20260925 05:04:21.921205 14762 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:04:21.921473 14762 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:04:21.924883 14762 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3: Sending DeleteTablet for 3 replicas of tablet b387151ce9a54e92b15d2d450b8d9bbb
I20260925 05:04:21.925881 15154 tablet_service.cc:1514] Processing DeleteTablet for tablet b387151ce9a54e92b15d2d450b8d9bbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:04:21 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:43774
I20260925 05:04:21.926066 15021 tablet_service.cc:1514] Processing DeleteTablet for tablet b387151ce9a54e92b15d2d450b8d9bbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:04:21 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:50156
I20260925 05:04:21.926631 14887 tablet_service.cc:1514] Processing DeleteTablet for tablet b387151ce9a54e92b15d2d450b8d9bbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:04:21 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:56466
I20260925 05:04:21.927011 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 14824
I20260925 05:04:21.937067 15262 tablet_replica.cc:330] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491: stopping tablet replica
I20260925 05:04:21.937536 15262 raft_consensus.cc:2229] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:04:21.938301 15262 raft_consensus.cc:2258] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:04:21.940474 15262 ts_tablet_manager.cc:1905] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:04:21.944293 15261 tablet_replica.cc:330] T b387151ce9a54e92b15d2d450b8d9bbb P 501237da7c2c4854b627f5015f66fb70: stopping tablet replica
I20260925 05:04:21.944777 15261 raft_consensus.cc:2229] T b387151ce9a54e92b15d2d450b8d9bbb P 501237da7c2c4854b627f5015f66fb70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:04:21.945475 15261 raft_consensus.cc:2258] T b387151ce9a54e92b15d2d450b8d9bbb P 501237da7c2c4854b627f5015f66fb70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:04:21.947403 15263 tablet_replica.cc:330] T b387151ce9a54e92b15d2d450b8d9bbb P 2efd1e14d8494b48b714b28b1476fb52: stopping tablet replica
I20260925 05:04:21.947780 15263 raft_consensus.cc:2229] T b387151ce9a54e92b15d2d450b8d9bbb P 2efd1e14d8494b48b714b28b1476fb52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:04:21.948235 15263 raft_consensus.cc:2258] T b387151ce9a54e92b15d2d450b8d9bbb P 2efd1e14d8494b48b714b28b1476fb52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:04:21.950130 15261 ts_tablet_manager.cc:1905] T b387151ce9a54e92b15d2d450b8d9bbb P 501237da7c2c4854b627f5015f66fb70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:04:21.950822 15262 ts_tablet_manager.cc:1918] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:04:21.950984 15262 log.cc:1196] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/wal/wals/b387151ce9a54e92b15d2d450b8d9bbb
I20260925 05:04:21.951489 15262 ts_tablet_manager.cc:1939] T b387151ce9a54e92b15d2d450b8d9bbb P 46c2a23747a5451ab3e440a2b7549491: Deleting consensus metadata
I20260925 05:04:21.953104 14747 catalog_manager.cc:4841] TS 46c2a23747a5451ab3e440a2b7549491 (127.30.247.2:46345): tablet b387151ce9a54e92b15d2d450b8d9bbb (table test-table [id=9ec86c42359047898994a32663057b90]) successfully deleted
I20260925 05:04:21.955626 15261 ts_tablet_manager.cc:1918] T b387151ce9a54e92b15d2d450b8d9bbb P 501237da7c2c4854b627f5015f66fb70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:04:21.955781 15261 log.cc:1196] T b387151ce9a54e92b15d2d450b8d9bbb P 501237da7c2c4854b627f5015f66fb70: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/wal/wals/b387151ce9a54e92b15d2d450b8d9bbb
I20260925 05:04:21.956802 15261 ts_tablet_manager.cc:1939] T b387151ce9a54e92b15d2d450b8d9bbb P 501237da7c2c4854b627f5015f66fb70: Deleting consensus metadata
I20260925 05:04:21.958096 15263 ts_tablet_manager.cc:1905] T b387151ce9a54e92b15d2d450b8d9bbb P 2efd1e14d8494b48b714b28b1476fb52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:04:21.958461 14750 catalog_manager.cc:4841] TS 501237da7c2c4854b627f5015f66fb70 (127.30.247.3:35489): tablet b387151ce9a54e92b15d2d450b8d9bbb (table test-table [id=9ec86c42359047898994a32663057b90]) successfully deleted
I20260925 05:04:21.965683 15263 ts_tablet_manager.cc:1918] T b387151ce9a54e92b15d2d450b8d9bbb P 2efd1e14d8494b48b714b28b1476fb52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:04:21.965831 15263 log.cc:1196] T b387151ce9a54e92b15d2d450b8d9bbb P 2efd1e14d8494b48b714b28b1476fb52: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/wal/wals/b387151ce9a54e92b15d2d450b8d9bbb
I20260925 05:04:21.966352 15263 ts_tablet_manager.cc:1939] T b387151ce9a54e92b15d2d450b8d9bbb P 2efd1e14d8494b48b714b28b1476fb52: Deleting consensus metadata
I20260925 05:04:21.968122 14750 catalog_manager.cc:4841] TS 2efd1e14d8494b48b714b28b1476fb52 (127.30.247.1:36567): tablet b387151ce9a54e92b15d2d450b8d9bbb (table test-table [id=9ec86c42359047898994a32663057b90]) successfully deleted
Sep 25 05:04:21 dist-test-slave-5n86 krb5kdc[14718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.1@KRBTEST.COM
I20260925 05:04:21.976810 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:21.927855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56476 (local address 127.30.247.1:36567)
0925 05:04:21.928063 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:21.928071 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:21.956431 (+ 28360us) server_negotiation.cc:407] Connection header received
0925 05:04:21.956791 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:21.956799 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:21.956885 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:21.956976 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:04:21.959367 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.960591 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.963395 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:21.964215 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:21.964309 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:21.970025 (+ 5716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:04:21.970040 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:04:21.970058 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:04:21.970097 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:04:21.973896 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:21.974612 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:21.974621 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:21.974625 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:21.974786 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:21.975197 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:21.975202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:21.975205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:21.975628 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:04:21.975939 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:21.976178 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 05:04:21.976401 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 05:04:21.977563 14947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:04:22.067329 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 14824
I20260925 05:04:22.087309 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 14958
Sep 25 05:04:22 dist-test-slave-5n86 krb5kdc[14718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.2@KRBTEST.COM
I20260925 05:04:22.105726 15234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:22.088133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50174 (local address 127.30.247.2:46345)
0925 05:04:22.088313 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:22.088317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:22.088415 (+ 98us) server_negotiation.cc:407] Connection header received
0925 05:04:22.088623 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:22.088634 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:22.088708 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:22.088794 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:04:22.090566 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:22.091403 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:22.093381 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:22.093883 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:22.093982 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:22.098905 (+ 4923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:04:22.098919 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:04:22.098946 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:04:22.098991 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:04:22.102904 (+ 3913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:22.103554 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:22.103563 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:22.103568 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:22.103731 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:22.104125 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:22.104130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:22.104133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:22.104593 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:04:22.104891 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:22.105125 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 05:04:22.105343 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 05:04:22.106446 15081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:04:22.219645 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 14958
I20260925 05:04:22.234555 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 15091
Sep 25 05:04:22 dist-test-slave-5n86 krb5kdc[14718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.3@KRBTEST.COM
I20260925 05:04:22.249341 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:22.235261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43790 (local address 127.30.247.3:35489)
0925 05:04:22.235445 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:22.235449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:22.235554 (+ 105us) server_negotiation.cc:407] Connection header received
0925 05:04:22.235717 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:22.235720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:22.235790 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:22.235866 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:04:22.237255 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:22.237984 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:22.239341 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:22.239725 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:22.239816 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:22.244140 (+ 4324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:04:22.244158 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:04:22.244178 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:04:22.244215 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:04:22.246964 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:22.247549 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:22.247556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:22.247558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:22.247678 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:22.247983 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:22.247986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:22.247987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:22.248310 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:04:22.248584 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:22.248933 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 05:04:22.249088 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 05:04:22.250023 15214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:04:22.337373 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 15091
I20260925 05:04:22.353791 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 14731
I20260925 05:04:22.355216 14793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:04:22.463618 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 14731
I20260925 05:04:42.485193 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:36613
--webserver_interface=127.30.247.62
--webserver_port=33827
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:36613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:04:42.818354 15269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:04:42.818691 15269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:04:42.818862 15269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:04:42.826141 15269 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 05:04:42.826220 15269 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 05:04:42.826287 15269 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 05:04:42.826363 15269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:04:42.826455 15269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:04:42.826565 15269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:04:42.826655 15269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:04:42.826745 15269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:04:42.836700 15269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:36613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:36613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=33827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:04:42.837926 15269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:04:42.839231 15269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:04:42.862612 15278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:04:42.862758 15269 server_base.cc:1034] running on GCE node
W20260925 05:04:42.862601 15275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:04:42.862603 15276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:04:42.864118 15269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:04:42.865895 15269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:04:42.867161 15269 hybrid_clock.cc:648] HybridClock initialized: now 1790312682867074 us; error 45 us; skew 500 ppm
I20260925 05:04:42.867594 15269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:04:42 dist-test-slave-5n86 krb5kdc[14718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312682, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:04:42.871266 15269 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:04:42.873006 15269 webserver.cc:460] Webserver started at http://127.30.247.62:33827/ using document root <none> and password file <none>
I20260925 05:04:42.873833 15269 fs_manager.cc:359] Metadata directory not provided
I20260925 05:04:42.873991 15269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:04:42.880447 15269 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 05:04:42.883420 15285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:42.884884 15269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 05:04:42.885070 15269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "f67d8ca88d4746afb89eeb62452ee1b3"
format_stamp: "Formatted at 2026-09-25 05:04:19 on dist-test-slave-5n86"
server_key: "6ebfe494484519838944b998fdb231bc"
server_key_iv: "cfe27d07e3a4248e42fe2e148cd65c0e"
server_key_version: "encryptionkey@0"
I20260925 05:04:42.886111 15269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:04:42.899765 15269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:04:42.903842 15269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:04:42.904142 15269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:04:42.923830 15269 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:36613
I20260925 05:04:42.923854 15336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:36613 every 8 connection(s)
I20260925 05:04:42.925493 15269 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:04:42.927790 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 15269
I20260925 05:04:42.928840 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:36567
--local_ip_for_outbound_sockets=127.30.247.1
--tserver_master_addrs=127.30.247.62:36613
--webserver_port=41357
--webserver_interface=127.30.247.1
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:04:42.935216 15337 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:04:42.940371 15337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3: Bootstrap starting.
I20260925 05:04:42.960718 15337 log.cc:824] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3: Log is configured to *not* fsync() on all Append() calls
I20260925 05:04:42.972949 15337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:04:42.973490 15337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3: Bootstrap complete.
I20260925 05:04:42.982705 15337 raft_consensus.cc:348] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d8ca88d4746afb89eeb62452ee1b3" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36613 } }
I20260925 05:04:42.983506 15337 raft_consensus.cc:729] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f67d8ca88d4746afb89eeb62452ee1b3, State: Initialized, Role: FOLLOWER
I20260925 05:04:42.984146 15337 consensus_queue.cc:260] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d8ca88d4746afb89eeb62452ee1b3" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36613 } }
I20260925 05:04:42.984382 15337 raft_consensus.cc:388] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:04:42.984501 15337 raft_consensus.cc:482] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:04:42.984711 15337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [term 1 FOLLOWER]: Advancing to term 2
I20260925 05:04:42.988384 15337 raft_consensus.cc:504] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d8ca88d4746afb89eeb62452ee1b3" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36613 } }
I20260925 05:04:42.988886 15337 leader_election.cc:302] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f67d8ca88d4746afb89eeb62452ee1b3; no voters:
I20260925 05:04:42.989557 15337 leader_election.cc:288] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 05:04:42.989718 15341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 05:04:42.991963 15337 sys_catalog.cc:564] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:04:42.992817 15341 raft_consensus.cc:686] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [term 2 LEADER]: Becoming Leader. State: Replica: f67d8ca88d4746afb89eeb62452ee1b3, State: Running, Role: LEADER
I20260925 05:04:42.993712 15341 consensus_queue.cc:237] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d8ca88d4746afb89eeb62452ee1b3" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36613 } }
I20260925 05:04:42.996439 15343 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f67d8ca88d4746afb89eeb62452ee1b3. Latest consensus state: current_term: 2 leader_uuid: "f67d8ca88d4746afb89eeb62452ee1b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d8ca88d4746afb89eeb62452ee1b3" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36613 } } }
I20260925 05:04:42.996809 15343 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [sys.catalog]: This master's current role is: LEADER
I20260925 05:04:42.996520 15342 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f67d8ca88d4746afb89eeb62452ee1b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d8ca88d4746afb89eeb62452ee1b3" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36613 } } }
I20260925 05:04:42.996964 15342 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [sys.catalog]: This master's current role is: LEADER
I20260925 05:04:43.009491 15350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:04:43.017678 15350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:04:43.019145 15350 catalog_manager.cc:1259] Loaded cluster ID: b082fe482e694e25915b0a16eb40d143
I20260925 05:04:43.019294 15350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:04:43.023150 15350 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:04:43.024701 15350 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3: Loaded TSK: 0
I20260925 05:04:43.024866 15358 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 05:04:43.054746 15350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3: Generated new TSK 1
I20260925 05:04:43.055222 15350 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 05:04:43.277879 15339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:04:43.278147 15339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:04:43.278290 15339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:04:43.285485 15339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:04:43.285584 15339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:04:43.285748 15339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:04:43.295785 15339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:36567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=41357
--tserver_master_addrs=127.30.247.62:36613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:04:43.297102 15339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:04:43.298381 15339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:04:43.312711 15364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:04:43.312687 15365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:04:43.313767 15339 server_base.cc:1034] running on GCE node
W20260925 05:04:43.313854 15367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:04:43.314599 15339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:04:43.315598 15339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:04:43.316855 15339 hybrid_clock.cc:648] HybridClock initialized: now 1790312683316799 us; error 46 us; skew 500 ppm
I20260925 05:04:43.317209 15339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:04:43 dist-test-slave-5n86 krb5kdc[14718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312683, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:04:43.320722 15339 init.cc:377] Logged in from keytab as kudu/127.30.247.1@KRBTEST.COM (short username kudu)
I20260925 05:04:43.322275 15339 webserver.cc:460] Webserver started at http://127.30.247.1:41357/ using document root <none> and password file <none>
I20260925 05:04:43.322959 15339 fs_manager.cc:359] Metadata directory not provided
I20260925 05:04:43.323119 15339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:04:43.328303 15339 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:04:43.331001 15374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:43.332377 15339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 05:04:43.332536 15339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "2efd1e14d8494b48b714b28b1476fb52"
format_stamp: "Formatted at 2026-09-25 05:04:20 on dist-test-slave-5n86"
server_key: "d13c5f23da204ecd2873a9dab89c50e0"
server_key_iv: "aee8b583550b09d80e2b38b3585601fa"
server_key_version: "encryptionkey@0"
I20260925 05:04:43.333504 15339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:04:43.356086 15339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:04:43.360127 15339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:04:43.360404 15339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:04:43.361508 15339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:04:43.362957 15339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:04:43.363070 15339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:43.363197 15339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:04:43.363281 15339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:43.396975 15339 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:36567
I20260925 05:04:43.397059 15486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:36567 every 8 connection(s)
I20260925 05:04:43.398408 15339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:04:43.405535 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 15339
I20260925 05:04:43.406366 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:46345
--local_ip_for_outbound_sockets=127.30.247.2
--tserver_master_addrs=127.30.247.62:36613
--webserver_port=32893
--webserver_interface=127.30.247.2
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:04:43 dist-test-slave-5n86 krb5kdc[14718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312683, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:04:43.424747 15490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:43.401590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.1:50903)
0925 05:04:43.402461 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:43.402509 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 05:04:43.404206 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:04:43.405629 (+ 1423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:04:43.405641 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:04:43.406444 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:04:43.407229 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:43.407259 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:43.409133 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:43.409142 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:43.411293 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:43.411311 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:43.412315 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:43.412324 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:43.412649 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:43.413405 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:04:43.413438 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:04:43.415672 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:04:43.420324 (+ 4652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:04:43.420331 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:04:43.420343 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:04:43.420853 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:04:43.421340 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:04:43.421343 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:04:43.421345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:04:43.421511 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:04:43.422502 (+ 991us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:04:43.422508 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:04:43.422847 (+ 339us) client_negotiation.cc:769] Sending connection context
0925 05:04:43.423128 (+ 281us) client_negotiation.cc:240] Negotiation successful
0925 05:04:43.423546 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":388,"threads_started":1}
I20260925 05:04:43.425097 15489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:43.401300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:50903 (local address 127.30.247.62:36613)
0925 05:04:43.401833 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:43.401846 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:43.402652 (+ 806us) server_negotiation.cc:407] Connection header received
0925 05:04:43.404729 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:43.404742 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:43.405081 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:43.405384 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:04:43.407452 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:43.408961 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:43.411441 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:43.412086 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:43.412215 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:43.415961 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:04:43.415973 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:04:43.415995 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:04:43.416045 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:04:43.420162 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:43.420985 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:43.420996 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:43.421007 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:43.421225 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:43.421627 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:43.421635 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:43.421641 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:43.422319 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:04:43.422498 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:43.423375 (+ 877us) server_negotiation.cc:299] Negotiation successful
0925 05:04:43.423830 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":226,"threads_started":1}
I20260925 05:04:43.427994 15487 heartbeater.cc:347] Connected to a master server at 127.30.247.62:36613
I20260925 05:04:43.428324 15487 heartbeater.cc:464] Registering TS with master...
I20260925 05:04:43.429127 15487 heartbeater.cc:511] Master 127.30.247.62:36613 requested a full tablet report, sending...
I20260925 05:04:43.431355 15302 ts_manager.cc:194] Registered new tserver with Master: 2efd1e14d8494b48b714b28b1476fb52 (127.30.247.1:36567)
I20260925 05:04:43.433059 15302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.1@KRBTEST.COM'} at 127.30.247.1:50903
W20260925 05:04:43.730724 15491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:04:43.731051 15491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:04:43.731230 15491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:04:43.738451 15491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:04:43.738601 15491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:04:43.738775 15491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:04:43.748447 15491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:46345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=32893
--tserver_master_addrs=127.30.247.62:36613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:04:43.749719 15491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:04:43.751011 15491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:04:43.763680 15499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:04:43.763758 15497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:04:43.763871 15496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:04:43.764838 15491 server_base.cc:1034] running on GCE node
I20260925 05:04:43.765400 15491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:04:43.766230 15491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:04:43.767417 15491 hybrid_clock.cc:648] HybridClock initialized: now 1790312683767362 us; error 49 us; skew 500 ppm
I20260925 05:04:43.767711 15491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:04:43 dist-test-slave-5n86 krb5kdc[14718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312683, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:04:43.770748 15491 init.cc:377] Logged in from keytab as kudu/127.30.247.2@KRBTEST.COM (short username kudu)
I20260925 05:04:43.772178 15491 webserver.cc:460] Webserver started at http://127.30.247.2:32893/ using document root <none> and password file <none>
I20260925 05:04:43.772902 15491 fs_manager.cc:359] Metadata directory not provided
I20260925 05:04:43.773061 15491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:04:43.778086 15491 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 05:04:43.780725 15506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:43.781988 15491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:04:43.782169 15491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "46c2a23747a5451ab3e440a2b7549491"
format_stamp: "Formatted at 2026-09-25 05:04:21 on dist-test-slave-5n86"
server_key: "c6ac0cf3e4ac96cb8852b7eccd2e9ba9"
server_key_iv: "e334f34cc78c0d54a3e477c2ec9bf809"
server_key_version: "encryptionkey@0"
I20260925 05:04:43.783119 15491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:04:43.806203 15491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:04:43.810333 15491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:04:43.810601 15491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:04:43.811540 15491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:04:43.812966 15491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:04:43.813089 15491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:43.813225 15491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:04:43.813314 15491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:43.844139 15491 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:46345
I20260925 05:04:43.844225 15618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:46345 every 8 connection(s)
I20260925 05:04:43.845683 15491 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:04:43.853173 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 15491
I20260925 05:04:43.853809 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:35489
--local_ip_for_outbound_sockets=127.30.247.3
--tserver_master_addrs=127.30.247.62:36613
--webserver_port=34513
--webserver_interface=127.30.247.3
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:04:43 dist-test-slave-5n86 krb5kdc[14718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312683, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:04:43.865751 15489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:43.848178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:36643 (local address 127.30.247.62:36613)
0925 05:04:43.848389 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:43.848396 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:43.849563 (+ 1167us) server_negotiation.cc:407] Connection header received
0925 05:04:43.851344 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:43.851348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:43.851417 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:43.851511 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:04:43.852985 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:43.853748 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:43.855331 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:43.855748 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:43.855885 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:43.859704 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:04:43.859713 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:04:43.859721 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:04:43.859770 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:04:43.862505 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:43.863440 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:43.863444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:43.863446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:43.863562 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:43.864036 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:43.864039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:43.864041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:43.864303 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:04:43.864412 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:43.865316 (+ 904us) server_negotiation.cc:299] Negotiation successful
0925 05:04:43.865483 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 05:04:43.867259 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:43.848706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.2:36643)
0925 05:04:43.849414 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:43.849454 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 05:04:43.851086 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:04:43.851699 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:04:43.851707 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:04:43.852211 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:04:43.852818 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:43.852836 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:43.853897 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:43.853904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:43.855186 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:43.855198 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:43.855966 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:43.855972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:43.856211 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:43.856972 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:04:43.857006 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:04:43.859436 (+ 2430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:04:43.862647 (+ 3211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:04:43.862658 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:04:43.862678 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:04:43.863307 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:04:43.863676 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:04:43.863683 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:04:43.863690 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:04:43.863924 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:04:43.864437 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:04:43.864448 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:04:43.864919 (+ 471us) client_negotiation.cc:769] Sending connection context
0925 05:04:43.865206 (+ 287us) client_negotiation.cc:240] Negotiation successful
0925 05:04:43.865768 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":280,"threads_started":1}
I20260925 05:04:43.868834 15619 heartbeater.cc:347] Connected to a master server at 127.30.247.62:36613
I20260925 05:04:43.869159 15619 heartbeater.cc:464] Registering TS with master...
I20260925 05:04:43.869865 15619 heartbeater.cc:511] Master 127.30.247.62:36613 requested a full tablet report, sending...
I20260925 05:04:43.871330 15302 ts_manager.cc:194] Registered new tserver with Master: 46c2a23747a5451ab3e440a2b7549491 (127.30.247.2:46345)
I20260925 05:04:43.872114 15302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.2@KRBTEST.COM'} at 127.30.247.2:36643
W20260925 05:04:44.191354 15622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:04:44.191648 15622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:04:44.191828 15622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:04:44.199137 15622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:04:44.199285 15622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:04:44.199472 15622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:04:44.208789 15622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:35489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=34513
--tserver_master_addrs=127.30.247.62:36613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:04:44.210002 15622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:04:44.211297 15622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:04:44.224334 15627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:04:44.224431 15628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:04:44.225047 15630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:04:44.225335 15622 server_base.cc:1034] running on GCE node
I20260925 05:04:44.225947 15622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:04:44.226847 15622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:04:44.228070 15622 hybrid_clock.cc:648] HybridClock initialized: now 1790312684228024 us; error 68 us; skew 500 ppm
I20260925 05:04:44.228377 15622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:04:44 dist-test-slave-5n86 krb5kdc[14718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312684, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:04:44.231463 15622 init.cc:377] Logged in from keytab as kudu/127.30.247.3@KRBTEST.COM (short username kudu)
I20260925 05:04:44.232872 15622 webserver.cc:460] Webserver started at http://127.30.247.3:34513/ using document root <none> and password file <none>
I20260925 05:04:44.233525 15622 fs_manager.cc:359] Metadata directory not provided
I20260925 05:04:44.233687 15622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:04:44.239187 15622 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 05:04:44.241776 15637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:44.243073 15622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:04:44.243239 15622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "501237da7c2c4854b627f5015f66fb70"
format_stamp: "Formatted at 2026-09-25 05:04:21 on dist-test-slave-5n86"
server_key: "e19eb6e09693392c367bc2b032278067"
server_key_iv: "4470128858c23210cbf297db32f12ee7"
server_key_version: "encryptionkey@0"
I20260925 05:04:44.244176 15622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:04:44.261489 15622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:04:44.265918 15622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:04:44.266196 15622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:04:44.267169 15622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:04:44.268474 15622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:04:44.268625 15622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:44.268754 15622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:04:44.268843 15622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:04:44.301411 15622 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:35489
I20260925 05:04:44.301494 15749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:35489 every 8 connection(s)
I20260925 05:04:44.302901 15622 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:04:44.309155 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 15622
Sep 25 05:04:44 dist-test-slave-5n86 krb5kdc[14718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312684, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:04:44.324345 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:44.310145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40008 (local address 127.30.247.62:36613)
0925 05:04:44.310591 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:44.310599 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:44.310910 (+ 311us) server_negotiation.cc:407] Connection header received
0925 05:04:44.311117 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:44.311129 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:44.311192 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:44.311300 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:04:44.312847 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.313578 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.314929 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.315362 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.315505 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:44.318761 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:04:44.318775 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:04:44.318779 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:04:44.318820 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:04:44.321564 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:44.322238 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:44.322249 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:44.322253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:44.322377 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:44.322774 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:44.322779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:44.322784 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:44.323056 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:04:44.323350 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:44.323658 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 05:04:44.323888 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":239,"threads_started":1}
I20260925 05:04:44.324470 15489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:44.305537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:47579 (local address 127.30.247.62:36613)
0925 05:04:44.305762 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:44.305768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:44.306914 (+ 1146us) server_negotiation.cc:407] Connection header received
0925 05:04:44.308933 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:44.308942 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:44.309040 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:44.309180 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:04:44.311079 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.311864 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.313478 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.313926 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.314061 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:44.317800 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:04:44.317808 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:04:44.317814 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:04:44.317859 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:04:44.321402 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:44.322147 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:44.322151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:44.322154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:44.322266 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:44.322706 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:44.322711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:44.322713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:44.323015 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:04:44.323203 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:44.323892 (+ 689us) server_negotiation.cc:299] Negotiation successful
0925 05:04:44.324117 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 05:04:44.325726 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:44.306007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.3:47579)
0925 05:04:44.306753 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:44.306796 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 05:04:44.308650 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:04:44.309415 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:04:44.309423 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:04:44.310081 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:04:44.310893 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:44.310913 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.312026 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.312036 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:44.313326 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:44.313338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.314156 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.314163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:44.314403 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:44.315066 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:04:44.315088 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:04:44.317547 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:04:44.321546 (+ 3999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:04:44.321568 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:04:44.321585 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:04:44.322005 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:04:44.322379 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:04:44.322385 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:04:44.322389 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:04:44.322580 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:04:44.323168 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:04:44.323179 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:04:44.323541 (+ 362us) client_negotiation.cc:769] Sending connection context
0925 05:04:44.323803 (+ 262us) client_negotiation.cc:240] Negotiation successful
0925 05:04:44.324329 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":304,"threads_started":1}
I20260925 05:04:44.327052 15750 heartbeater.cc:347] Connected to a master server at 127.30.247.62:36613
I20260925 05:04:44.327354 15750 heartbeater.cc:464] Registering TS with master...
I20260925 05:04:44.328045 15750 heartbeater.cc:511] Master 127.30.247.62:36613 requested a full tablet report, sending...
I20260925 05:04:44.329396 15302 ts_manager.cc:194] Registered new tserver with Master: 501237da7c2c4854b627f5015f66fb70 (127.30.247.3:35489)
I20260925 05:04:44.330164 15302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.3@KRBTEST.COM'} at 127.30.247.3:47579
I20260925 05:04:44.339460 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:04:44.346408 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:44.340400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40020 (local address 127.30.247.62:36613)
0925 05:04:44.340635 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:44.340643 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:44.341198 (+ 555us) server_negotiation.cc:407] Connection header received
0925 05:04:44.341413 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:44.341417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:44.341554 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:44.341684 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:04:44.342122 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.342847 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.344435 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.344996 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.345158 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:44.345641 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:04:44.345854 (+ 213us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 05:04:44.346153 (+ 299us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40020: token expired
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 05:04:44.346540 15753 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40020: token expired
W20260925 05:04:44.347065 15225 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.30.247.62:36613)
I20260925 05:04:44.360417 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:44.347752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40032 (local address 127.30.247.62:36613)
0925 05:04:44.347900 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:44.347905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:44.347919 (+ 14us) server_negotiation.cc:407] Connection header received
0925 05:04:44.348017 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:44.348021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:44.348075 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:44.348148 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:04:44.349742 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.350532 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.351864 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.352255 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.352356 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:44.355141 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:04:44.355157 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:04:44.355160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:04:44.355193 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:04:44.357807 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:44.358475 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:44.358479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:44.358481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:44.358617 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:44.359002 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:44.359005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:44.359007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:44.359332 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:04:44.359513 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:44.359850 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 05:04:44.360028 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 05:04:44.363934 15302 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:04:44.371270 15302 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:04:44.403425 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:44.383854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33128 (local address 127.30.247.3:35489)
0925 05:04:44.393150 (+ 9296us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:44.393163 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:44.394745 (+ 1582us) server_negotiation.cc:407] Connection header received
0925 05:04:44.395032 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:44.395041 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:44.395275 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:44.395437 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:04:44.396377 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.397638 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.400602 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.402003 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.402128 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:44.402342 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:44.402948 (+ 606us) server_negotiation.cc:299] Negotiation successful
0925 05:04:44.403114 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9103,"thread_start_us":8942,"threads_started":1}
I20260925 05:04:44.404608 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:44.383454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:35489 (local address 127.0.0.1:33128)
0925 05:04:44.394634 (+ 11180us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:44.394652 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:04:44.394850 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:04:44.395682 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:04:44.395687 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:04:44.395715 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:04:44.396160 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:44.396176 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.397868 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.397875 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:44.400397 (+ 2522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:44.400417 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.402497 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.402504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:44.402686 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:44.402736 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 05:04:44.403917 (+ 1181us) client_negotiation.cc:240] Negotiation successful
0925 05:04:44.404127 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10859,"mutex_wait_us":3,"thread_start_us":10720,"threads_started":1}
I20260925 05:04:44.405184 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:44.383094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56770 (local address 127.30.247.2:46345)
0925 05:04:44.394317 (+ 11223us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:44.394325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:44.394355 (+ 30us) server_negotiation.cc:407] Connection header received
0925 05:04:44.394463 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:44.394471 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:44.394738 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:44.394939 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:04:44.395664 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.399031 (+ 3367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.401993 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.403760 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.403856 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:44.404030 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:44.404699 (+ 669us) server_negotiation.cc:299] Negotiation successful
0925 05:04:44.404864 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11046,"thread_start_us":11359,"threads_started":1}
I20260925 05:04:44.405531 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:44.395564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:36567 (local address 127.0.0.1:43744)
0925 05:04:44.396127 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:44.396148 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:04:44.396270 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:04:44.397081 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:04:44.397086 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:04:44.397110 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:04:44.397490 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:44.397503 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.399562 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.399571 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:44.402388 (+ 2817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:44.402402 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.404702 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.404708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:44.404886 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:44.404920 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 05:04:44.405014 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 05:04:44.405152 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":626,"thread_start_us":278,"threads_started":1}
I20260925 05:04:44.405937 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:44.382890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:46345 (local address 127.0.0.1:56770)
0925 05:04:44.393540 (+ 10650us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:44.393577 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:04:44.393734 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:04:44.394981 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:04:44.394993 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:04:44.395039 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:04:44.395449 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:44.395465 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.399212 (+ 3747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.399217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:44.401814 (+ 2597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:44.401829 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.404222 (+ 2393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.404229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:44.404431 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:44.404467 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 05:04:44.405439 (+ 972us) client_negotiation.cc:240] Negotiation successful
0925 05:04:44.405566 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10413,"mutex_wait_us":42,"thread_start_us":10292,"threads_started":1}
I20260925 05:04:44.406319 15764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:44.394771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43744 (local address 127.30.247.1:36567)
0925 05:04:44.395556 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:44.395564 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:44.396248 (+ 684us) server_negotiation.cc:407] Connection header received
0925 05:04:44.396427 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:44.396437 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:44.396678 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:44.396854 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:04:44.397690 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.399382 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.402568 (+ 3186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.404437 (+ 1869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.404620 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:44.404841 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:44.405820 (+ 979us) server_negotiation.cc:299] Negotiation successful
0925 05:04:44.405998 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":504,"threads_started":1}
I20260925 05:04:44.408640 15685 tablet_service.cc:1467] Processing CreateTablet for tablet 003ad3c426044380abda69d1b1a44d49 (DEFAULT_TABLE table=test-table [id=158e18803aef4c229d488946206dc19e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:04:44.408953 15554 tablet_service.cc:1467] Processing CreateTablet for tablet 003ad3c426044380abda69d1b1a44d49 (DEFAULT_TABLE table=test-table [id=158e18803aef4c229d488946206dc19e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:04:44.410065 15685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 003ad3c426044380abda69d1b1a44d49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:04:44.410331 15554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 003ad3c426044380abda69d1b1a44d49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:04:44.412068 15422 tablet_service.cc:1467] Processing CreateTablet for tablet 003ad3c426044380abda69d1b1a44d49 (DEFAULT_TABLE table=test-table [id=158e18803aef4c229d488946206dc19e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:04:44.413404 15422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 003ad3c426044380abda69d1b1a44d49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:04:44.422298 15766 tablet_bootstrap.cc:492] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70: Bootstrap starting.
I20260925 05:04:44.425060 15767 tablet_bootstrap.cc:492] T 003ad3c426044380abda69d1b1a44d49 P 46c2a23747a5451ab3e440a2b7549491: Bootstrap starting.
I20260925 05:04:44.425657 15766 tablet_bootstrap.cc:654] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70: Neither blocks nor log segments found. Creating new log.
I20260925 05:04:44.427520 15766 log.cc:824] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70: Log is configured to *not* fsync() on all Append() calls
I20260925 05:04:44.428143 15768 tablet_bootstrap.cc:492] T 003ad3c426044380abda69d1b1a44d49 P 2efd1e14d8494b48b714b28b1476fb52: Bootstrap starting.
I20260925 05:04:44.428579 15767 tablet_bootstrap.cc:654] T 003ad3c426044380abda69d1b1a44d49 P 46c2a23747a5451ab3e440a2b7549491: Neither blocks nor log segments found. Creating new log.
I20260925 05:04:44.430160 15767 log.cc:824] T 003ad3c426044380abda69d1b1a44d49 P 46c2a23747a5451ab3e440a2b7549491: Log is configured to *not* fsync() on all Append() calls
I20260925 05:04:44.430961 15768 tablet_bootstrap.cc:654] T 003ad3c426044380abda69d1b1a44d49 P 2efd1e14d8494b48b714b28b1476fb52: Neither blocks nor log segments found. Creating new log.
I20260925 05:04:44.432127 15768 log.cc:824] T 003ad3c426044380abda69d1b1a44d49 P 2efd1e14d8494b48b714b28b1476fb52: Log is configured to *not* fsync() on all Append() calls
I20260925 05:04:44.432850 15766 tablet_bootstrap.cc:492] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70: No bootstrap required, opened a new log
I20260925 05:04:44.433142 15766 ts_tablet_manager.cc:1397] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260925 05:04:44.433311 15767 tablet_bootstrap.cc:492] T 003ad3c426044380abda69d1b1a44d49 P 46c2a23747a5451ab3e440a2b7549491: No bootstrap required, opened a new log
I20260925 05:04:44.433521 15767 ts_tablet_manager.cc:1397] T 003ad3c426044380abda69d1b1a44d49 P 46c2a23747a5451ab3e440a2b7549491: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260925 05:04:44.435153 15768 tablet_bootstrap.cc:492] T 003ad3c426044380abda69d1b1a44d49 P 2efd1e14d8494b48b714b28b1476fb52: No bootstrap required, opened a new log
I20260925 05:04:44.435424 15768 ts_tablet_manager.cc:1397] T 003ad3c426044380abda69d1b1a44d49 P 2efd1e14d8494b48b714b28b1476fb52: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260925 05:04:44.436146 15487 heartbeater.cc:503] Master 127.30.247.62:36613 was elected leader, sending a full tablet report...
I20260925 05:04:44.442312 15766 raft_consensus.cc:348] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c2a23747a5451ab3e440a2b7549491" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46345 } } peers { permanent_uuid: "501237da7c2c4854b627f5015f66fb70" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35489 } } peers { permanent_uuid: "2efd1e14d8494b48b714b28b1476fb52" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36567 } }
I20260925 05:04:44.442693 15766 raft_consensus.cc:374] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:04:44.442826 15766 raft_consensus.cc:729] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501237da7c2c4854b627f5015f66fb70, State: Initialized, Role: FOLLOWER
I20260925 05:04:44.443449 15766 consensus_queue.cc:260] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c2a23747a5451ab3e440a2b7549491" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46345 } } peers { permanent_uuid: "501237da7c2c4854b627f5015f66fb70" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35489 } } peers { permanent_uuid: "2efd1e14d8494b48b714b28b1476fb52" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36567 } }
I20260925 05:04:44.444561 15750 heartbeater.cc:503] Master 127.30.247.62:36613 was elected leader, sending a full tablet report...
I20260925 05:04:44.444476 15767 raft_consensus.cc:348] T 003ad3c426044380abda69d1b1a44d49 P 46c2a23747a5451ab3e440a2b7549491 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c2a23747a5451ab3e440a2b7549491" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46345 } } peers { permanent_uuid: "501237da7c2c4854b627f5015f66fb70" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35489 } } peers { permanent_uuid: "2efd1e14d8494b48b714b28b1476fb52" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36567 } }
I20260925 05:04:44.444908 15767 raft_consensus.cc:374] T 003ad3c426044380abda69d1b1a44d49 P 46c2a23747a5451ab3e440a2b7549491 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:04:44.444990 15767 raft_consensus.cc:729] T 003ad3c426044380abda69d1b1a44d49 P 46c2a23747a5451ab3e440a2b7549491 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c2a23747a5451ab3e440a2b7549491, State: Initialized, Role: FOLLOWER
I20260925 05:04:44.445325 15766 ts_tablet_manager.cc:1428] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260925 05:04:44.445573 15767 consensus_queue.cc:260] T 003ad3c426044380abda69d1b1a44d49 P 46c2a23747a5451ab3e440a2b7549491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c2a23747a5451ab3e440a2b7549491" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46345 } } peers { permanent_uuid: "501237da7c2c4854b627f5015f66fb70" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35489 } } peers { permanent_uuid: "2efd1e14d8494b48b714b28b1476fb52" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36567 } }
I20260925 05:04:44.446116 15768 raft_consensus.cc:348] T 003ad3c426044380abda69d1b1a44d49 P 2efd1e14d8494b48b714b28b1476fb52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c2a23747a5451ab3e440a2b7549491" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46345 } } peers { permanent_uuid: "501237da7c2c4854b627f5015f66fb70" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35489 } } peers { permanent_uuid: "2efd1e14d8494b48b714b28b1476fb52" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36567 } }
I20260925 05:04:44.446606 15768 raft_consensus.cc:374] T 003ad3c426044380abda69d1b1a44d49 P 2efd1e14d8494b48b714b28b1476fb52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:04:44.446727 15768 raft_consensus.cc:729] T 003ad3c426044380abda69d1b1a44d49 P 2efd1e14d8494b48b714b28b1476fb52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efd1e14d8494b48b714b28b1476fb52, State: Initialized, Role: FOLLOWER
I20260925 05:04:44.447391 15768 consensus_queue.cc:260] T 003ad3c426044380abda69d1b1a44d49 P 2efd1e14d8494b48b714b28b1476fb52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c2a23747a5451ab3e440a2b7549491" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46345 } } peers { permanent_uuid: "501237da7c2c4854b627f5015f66fb70" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35489 } } peers { permanent_uuid: "2efd1e14d8494b48b714b28b1476fb52" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36567 } }
I20260925 05:04:44.449771 15768 ts_tablet_manager.cc:1428] T 003ad3c426044380abda69d1b1a44d49 P 2efd1e14d8494b48b714b28b1476fb52: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260925 05:04:44.451094 15619 heartbeater.cc:503] Master 127.30.247.62:36613 was elected leader, sending a full tablet report...
I20260925 05:04:44.451650 15767 ts_tablet_manager.cc:1428] T 003ad3c426044380abda69d1b1a44d49 P 46c2a23747a5451ab3e440a2b7549491: Time spent starting tablet: real 0.018s user 0.009s sys 0.008s
W20260925 05:04:44.554556 15751 tablet.cc:2369] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:04:44.597640 15620 tablet.cc:2369] T 003ad3c426044380abda69d1b1a44d49 P 46c2a23747a5451ab3e440a2b7549491: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:04:44.601522 15772 raft_consensus.cc:482] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:04:44.601854 15772 raft_consensus.cc:504] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c2a23747a5451ab3e440a2b7549491" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46345 } } peers { permanent_uuid: "501237da7c2c4854b627f5015f66fb70" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35489 } } peers { permanent_uuid: "2efd1e14d8494b48b714b28b1476fb52" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36567 } }
I20260925 05:04:44.603616 15772 leader_election.cc:288] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46c2a23747a5451ab3e440a2b7549491 (127.30.247.2:46345), 2efd1e14d8494b48b714b28b1476fb52 (127.30.247.1:36567)
I20260925 05:04:44.609298 15764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:44.604485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:39515 (local address 127.30.247.1:36567)
0925 05:04:44.604690 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:44.604695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:44.604710 (+ 15us) server_negotiation.cc:407] Connection header received
0925 05:04:44.604770 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:44.604774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:44.604829 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:44.604938 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:04:44.605322 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.606019 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.607613 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.608379 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.608582 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:44.608666 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:44.608967 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 05:04:44.609062 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 05:04:44.609349 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:44.603887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:36567 (local address 127.30.247.3:39515)
0925 05:04:44.604178 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:44.604194 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:04:44.604294 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:04:44.604954 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:04:44.604958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:04:44.604984 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:04:44.605169 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:44.605177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.606151 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.606158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:44.607442 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:44.607453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.608588 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.608595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:44.608785 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:44.608822 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 05:04:44.608917 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 05:04:44.609052 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260925 05:04:44.610102 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:44.603999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:46345 (local address 127.30.247.3:45765)
0925 05:04:44.604777 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0925 05:04:44.604792 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:04:44.604936 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:04:44.605268 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:04:44.605272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:04:44.605294 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:04:44.605622 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:44.605630 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.606600 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.606605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:44.607931 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:04:44.607943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.609338 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.609347 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:04:44.609524 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:44.609558 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 05:04:44.609647 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 05:04:44.609764 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":3,"thread_start_us":465,"threads_started":1}
I20260925 05:04:44.610275 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:44.603957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:45765 (local address 127.30.247.2:46345)
0925 05:04:44.604167 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:44.604173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:44.604884 (+ 711us) server_negotiation.cc:407] Connection header received
0925 05:04:44.605094 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:44.605098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:44.605156 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:44.605254 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:04:44.605772 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.606451 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.608095 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.609126 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.609270 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:44.609334 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:44.609956 (+ 622us) server_negotiation.cc:299] Negotiation successful
0925 05:04:44.610044 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 05:04:44.610486 15442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003ad3c426044380abda69d1b1a44d49" candidate_uuid: "501237da7c2c4854b627f5015f66fb70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efd1e14d8494b48b714b28b1476fb52" is_pre_election: true
I20260925 05:04:44.611054 15442 raft_consensus.cc:2454] T 003ad3c426044380abda69d1b1a44d49 P 2efd1e14d8494b48b714b28b1476fb52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 501237da7c2c4854b627f5015f66fb70 in term 0.
I20260925 05:04:44.611197 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003ad3c426044380abda69d1b1a44d49" candidate_uuid: "501237da7c2c4854b627f5015f66fb70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c2a23747a5451ab3e440a2b7549491" is_pre_election: true
I20260925 05:04:44.611722 15574 raft_consensus.cc:2454] T 003ad3c426044380abda69d1b1a44d49 P 46c2a23747a5451ab3e440a2b7549491 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 501237da7c2c4854b627f5015f66fb70 in term 0.
I20260925 05:04:44.611814 15641 leader_election.cc:302] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efd1e14d8494b48b714b28b1476fb52, 501237da7c2c4854b627f5015f66fb70; no voters:
I20260925 05:04:44.612280 15772 raft_consensus.cc:2781] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:04:44.612432 15772 raft_consensus.cc:482] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:04:44.612565 15772 raft_consensus.cc:3037] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:04:44.614677 15772 raft_consensus.cc:504] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c2a23747a5451ab3e440a2b7549491" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46345 } } peers { permanent_uuid: "501237da7c2c4854b627f5015f66fb70" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35489 } } peers { permanent_uuid: "2efd1e14d8494b48b714b28b1476fb52" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36567 } }
I20260925 05:04:44.615283 15772 leader_election.cc:288] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70 [CANDIDATE]: Term 1 election: Requested vote from peers 46c2a23747a5451ab3e440a2b7549491 (127.30.247.2:46345), 2efd1e14d8494b48b714b28b1476fb52 (127.30.247.1:36567)
I20260925 05:04:44.615653 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003ad3c426044380abda69d1b1a44d49" candidate_uuid: "501237da7c2c4854b627f5015f66fb70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c2a23747a5451ab3e440a2b7549491"
I20260925 05:04:44.615860 15574 raft_consensus.cc:3037] T 003ad3c426044380abda69d1b1a44d49 P 46c2a23747a5451ab3e440a2b7549491 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:04:44.615860 15442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003ad3c426044380abda69d1b1a44d49" candidate_uuid: "501237da7c2c4854b627f5015f66fb70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efd1e14d8494b48b714b28b1476fb52"
I20260925 05:04:44.616093 15442 raft_consensus.cc:3037] T 003ad3c426044380abda69d1b1a44d49 P 2efd1e14d8494b48b714b28b1476fb52 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:04:44.618083 15574 raft_consensus.cc:2454] T 003ad3c426044380abda69d1b1a44d49 P 46c2a23747a5451ab3e440a2b7549491 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 501237da7c2c4854b627f5015f66fb70 in term 1.
I20260925 05:04:44.618330 15442 raft_consensus.cc:2454] T 003ad3c426044380abda69d1b1a44d49 P 2efd1e14d8494b48b714b28b1476fb52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 501237da7c2c4854b627f5015f66fb70 in term 1.
I20260925 05:04:44.618466 15638 leader_election.cc:302] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46c2a23747a5451ab3e440a2b7549491, 501237da7c2c4854b627f5015f66fb70; no voters:
I20260925 05:04:44.618777 15772 raft_consensus.cc:2781] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:04:44.619211 15772 raft_consensus.cc:686] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70 [term 1 LEADER]: Becoming Leader. State: Replica: 501237da7c2c4854b627f5015f66fb70, State: Running, Role: LEADER
I20260925 05:04:44.619643 15772 consensus_queue.cc:237] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c2a23747a5451ab3e440a2b7549491" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46345 } } peers { permanent_uuid: "501237da7c2c4854b627f5015f66fb70" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35489 } } peers { permanent_uuid: "2efd1e14d8494b48b714b28b1476fb52" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36567 } }
I20260925 05:04:44.624647 15302 catalog_manager.cc:5495] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 501237da7c2c4854b627f5015f66fb70 (127.30.247.3). New cstate: current_term: 1 leader_uuid: "501237da7c2c4854b627f5015f66fb70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c2a23747a5451ab3e440a2b7549491" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501237da7c2c4854b627f5015f66fb70" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efd1e14d8494b48b714b28b1476fb52" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36567 } health_report { overall_health: UNKNOWN } } }
I20260925 05:04:44.645294 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:44.639972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33132 (local address 127.30.247.3:35489)
0925 05:04:44.640156 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:44.640160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:44.640178 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:04:44.640363 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:44.640367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:44.640437 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:44.640620 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:04:44.640938 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.641654 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.643010 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.643433 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.643530 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:44.643998 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:04:44.644712 (+ 714us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:04:44.644818 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:44.644933 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 05:04:44.644997 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 05:04:44.650671 15488 tablet.cc:2369] T 003ad3c426044380abda69d1b1a44d49 P 2efd1e14d8494b48b714b28b1476fb52: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:04:44.653215 15442 raft_consensus.cc:1261] T 003ad3c426044380abda69d1b1a44d49 P 2efd1e14d8494b48b714b28b1476fb52 [term 1 FOLLOWER]: Refusing update from remote peer 501237da7c2c4854b627f5015f66fb70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:04:44.653239 15574 raft_consensus.cc:1261] T 003ad3c426044380abda69d1b1a44d49 P 46c2a23747a5451ab3e440a2b7549491 [term 1 FOLLOWER]: Refusing update from remote peer 501237da7c2c4854b627f5015f66fb70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:04:44.654165 15776 consensus_queue.cc:1035] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efd1e14d8494b48b714b28b1476fb52" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 36567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:04:44.654501 15772 consensus_queue.cc:1035] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c2a23747a5451ab3e440a2b7549491" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 46345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:04:44.679016 15301 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:04:44.679345 15301 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:04:44.683118 15301 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3: Sending DeleteTablet for 3 replicas of tablet 003ad3c426044380abda69d1b1a44d49
I20260925 05:04:44.684316 15554 tablet_service.cc:1514] Processing DeleteTablet for tablet 003ad3c426044380abda69d1b1a44d49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:04:44 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:56770
I20260925 05:04:44.684327 15422 tablet_service.cc:1514] Processing DeleteTablet for tablet 003ad3c426044380abda69d1b1a44d49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:04:44 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:43744
I20260925 05:04:44.684412 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 15339
I20260925 05:04:44.685211 15685 tablet_service.cc:1514] Processing DeleteTablet for tablet 003ad3c426044380abda69d1b1a44d49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:04:44 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:33128
I20260925 05:04:44.688345 15789 tablet_replica.cc:330] T 003ad3c426044380abda69d1b1a44d49 P 46c2a23747a5451ab3e440a2b7549491: stopping tablet replica
I20260925 05:04:44.688639 15791 tablet_replica.cc:330] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70: stopping tablet replica
I20260925 05:04:44.688640 15790 tablet_replica.cc:330] T 003ad3c426044380abda69d1b1a44d49 P 2efd1e14d8494b48b714b28b1476fb52: stopping tablet replica
I20260925 05:04:44.688820 15789 raft_consensus.cc:2229] T 003ad3c426044380abda69d1b1a44d49 P 46c2a23747a5451ab3e440a2b7549491 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:04:44.689064 15791 raft_consensus.cc:2229] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:04:44.689069 15790 raft_consensus.cc:2229] T 003ad3c426044380abda69d1b1a44d49 P 2efd1e14d8494b48b714b28b1476fb52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:04:44.689342 15789 raft_consensus.cc:2258] T 003ad3c426044380abda69d1b1a44d49 P 46c2a23747a5451ab3e440a2b7549491 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:04:44.689504 15790 raft_consensus.cc:2258] T 003ad3c426044380abda69d1b1a44d49 P 2efd1e14d8494b48b714b28b1476fb52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:04:44.689743 15791 raft_consensus.cc:2258] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:04:44.691771 15790 ts_tablet_manager.cc:1905] T 003ad3c426044380abda69d1b1a44d49 P 2efd1e14d8494b48b714b28b1476fb52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:04:44.692782 15789 ts_tablet_manager.cc:1905] T 003ad3c426044380abda69d1b1a44d49 P 46c2a23747a5451ab3e440a2b7549491: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:04:44.693580 15791 ts_tablet_manager.cc:1905] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:04:44.699440 15790 ts_tablet_manager.cc:1918] T 003ad3c426044380abda69d1b1a44d49 P 2efd1e14d8494b48b714b28b1476fb52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:04:44.699657 15790 log.cc:1196] T 003ad3c426044380abda69d1b1a44d49 P 2efd1e14d8494b48b714b28b1476fb52: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/wal/wals/003ad3c426044380abda69d1b1a44d49
I20260925 05:04:44.700275 15789 ts_tablet_manager.cc:1918] T 003ad3c426044380abda69d1b1a44d49 P 46c2a23747a5451ab3e440a2b7549491: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:04:44.700372 15790 ts_tablet_manager.cc:1939] T 003ad3c426044380abda69d1b1a44d49 P 2efd1e14d8494b48b714b28b1476fb52: Deleting consensus metadata
I20260925 05:04:44.702550 15289 catalog_manager.cc:4841] TS 2efd1e14d8494b48b714b28b1476fb52 (127.30.247.1:36567): tablet 003ad3c426044380abda69d1b1a44d49 (table test-table [id=158e18803aef4c229d488946206dc19e]) successfully deleted
I20260925 05:04:44.702637 15789 log.cc:1196] T 003ad3c426044380abda69d1b1a44d49 P 46c2a23747a5451ab3e440a2b7549491: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/wal/wals/003ad3c426044380abda69d1b1a44d49
I20260925 05:04:44.703205 15789 ts_tablet_manager.cc:1939] T 003ad3c426044380abda69d1b1a44d49 P 46c2a23747a5451ab3e440a2b7549491: Deleting consensus metadata
I20260925 05:04:44.703938 15764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:44.685345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43746 (local address 127.30.247.1:36567)
0925 05:04:44.685503 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:44.685511 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:44.685534 (+ 23us) server_negotiation.cc:407] Connection header received
0925 05:04:44.685744 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:44.685750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:44.685873 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:44.685997 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:04:44.687841 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.689017 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.691075 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.691698 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.691831 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:44.695357 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:04:44.695370 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:04:44.695384 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:04:44.695426 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:04:44.699390 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:44.700485 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:44.700493 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:44.700496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:44.700665 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:44.701210 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:44.701215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:44.701218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:44.701783 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:04:44.702091 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:44.703341 (+ 1250us) server_negotiation.cc:299] Negotiation successful
0925 05:04:44.703566 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 05:04:44.704870 15286 catalog_manager.cc:4841] TS 46c2a23747a5451ab3e440a2b7549491 (127.30.247.2:46345): tablet 003ad3c426044380abda69d1b1a44d49 (table test-table [id=158e18803aef4c229d488946206dc19e]) successfully deleted
I20260925 05:04:44.704947 15482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:04:44.709240 15791 ts_tablet_manager.cc:1918] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:04:44.709407 15791 log.cc:1196] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/wal/wals/003ad3c426044380abda69d1b1a44d49
I20260925 05:04:44.709934 15791 ts_tablet_manager.cc:1939] T 003ad3c426044380abda69d1b1a44d49 P 501237da7c2c4854b627f5015f66fb70: Deleting consensus metadata
I20260925 05:04:44.711618 15289 catalog_manager.cc:4841] TS 501237da7c2c4854b627f5015f66fb70 (127.30.247.3:35489): tablet 003ad3c426044380abda69d1b1a44d49 (table test-table [id=158e18803aef4c229d488946206dc19e]) successfully deleted
I20260925 05:04:44.794090 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 15339
I20260925 05:04:44.812224 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 15491
I20260925 05:04:44.829928 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:44.813125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56776 (local address 127.30.247.2:46345)
0925 05:04:44.813311 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:44.813317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:44.813338 (+ 21us) server_negotiation.cc:407] Connection header received
0925 05:04:44.813458 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:44.813463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:44.813540 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:44.813657 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:04:44.814942 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.815706 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.817051 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.817455 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.817591 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:44.824077 (+ 6486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:04:44.824099 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:04:44.824114 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:04:44.824149 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:04:44.827054 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:44.827771 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:44.827781 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:44.827784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:44.827915 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:44.828306 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:44.828312 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:44.828315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:44.828762 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:04:44.829071 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:44.829364 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 05:04:44.829581 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 05:04:44.830685 15614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:04:44.921365 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 15491
I20260925 05:04:44.945988 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 15622
I20260925 05:04:44.963449 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:04:44.948016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33138 (local address 127.30.247.3:35489)
0925 05:04:44.948227 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 05:04:44.948233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:04:44.948279 (+ 46us) server_negotiation.cc:407] Connection header received
0925 05:04:44.948456 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:04:44.948463 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:04:44.948623 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:04:44.948782 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:04:44.950065 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.951296 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.952503 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:04:44.953540 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:04:44.953690 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:04:44.956211 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:04:44.956227 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:04:44.956247 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:04:44.956291 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:04:44.960418 (+ 4127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:44.961150 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:44.961161 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:44.961165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:44.961358 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:04:44.961704 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:04:44.961709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:04:44.961713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:04:44.962194 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:04:44.962509 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0925 05:04:44.962732 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 05:04:44.962967 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 05:04:44.964305 15745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:04:45.084918 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 15622
I20260925 05:04:45.105196 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 15269
I20260925 05:04:45.106309 15332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:04:45.220783 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 15269
I20260925 05:05:05.243124 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:36613
--webserver_interface=127.30.247.62
--webserver_port=33827
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:36613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:05.561353 15796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:05.561638 15796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:05.561812 15796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:05.569020 15796 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 05:05:05.569119 15796 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 05:05:05.569157 15796 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 05:05:05.569252 15796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:05.569340 15796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:05.569465 15796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:05.569545 15796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:05.569628 15796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:05.579365 15796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:36613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:36613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=33827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:05.580681 15796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:05.582015 15796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:05.594063 15801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:05.594162 15804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:05.594183 15802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:05.595073 15796 server_base.cc:1034] running on GCE node
I20260925 05:05:05.595834 15796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:05.597231 15796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:05.598506 15796 hybrid_clock.cc:648] HybridClock initialized: now 1790312705598429 us; error 52 us; skew 500 ppm
I20260925 05:05:05.598912 15796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:05:05 dist-test-slave-5n86 krb5kdc[14718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312705, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:05.602161 15796 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:05:05.603654 15796 webserver.cc:460] Webserver started at http://127.30.247.62:33827/ using document root <none> and password file <none>
I20260925 05:05:05.604341 15796 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:05.604482 15796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:05.609688 15796 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:05:05.612463 15811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:05.613759 15796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:05:05.613938 15796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "f67d8ca88d4746afb89eeb62452ee1b3"
format_stamp: "Formatted at 2026-09-25 05:04:19 on dist-test-slave-5n86"
server_key: "6ebfe494484519838944b998fdb231bc"
server_key_iv: "cfe27d07e3a4248e42fe2e148cd65c0e"
server_key_version: "encryptionkey@0"
I20260925 05:05:05.614886 15796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:05.630844 15796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:05.634794 15796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:05.635089 15796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:05.654521 15796 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:36613
I20260925 05:05:05.654563 15862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:36613 every 8 connection(s)
I20260925 05:05:05.655973 15796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:05.662659 15863 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:05:05.665759 15863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3: Bootstrap starting.
I20260925 05:05:05.666416 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 15796
I20260925 05:05:05.667416 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:36567
--local_ip_for_outbound_sockets=127.30.247.1
--tserver_master_addrs=127.30.247.62:36613
--webserver_port=41357
--webserver_interface=127.30.247.1
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:05:05.689145 15863 log.cc:824] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:05.704622 15863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:05:05.705152 15863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3: Bootstrap complete.
I20260925 05:05:05.714026 15863 raft_consensus.cc:348] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d8ca88d4746afb89eeb62452ee1b3" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36613 } }
I20260925 05:05:05.714833 15863 raft_consensus.cc:729] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f67d8ca88d4746afb89eeb62452ee1b3, State: Initialized, Role: FOLLOWER
I20260925 05:05:05.715524 15863 consensus_queue.cc:260] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d8ca88d4746afb89eeb62452ee1b3" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36613 } }
I20260925 05:05:05.715750 15863 raft_consensus.cc:388] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:05.715905 15863 raft_consensus.cc:482] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:05.716070 15863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [term 2 FOLLOWER]: Advancing to term 3
I20260925 05:05:05.720052 15863 raft_consensus.cc:504] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d8ca88d4746afb89eeb62452ee1b3" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36613 } }
I20260925 05:05:05.720516 15863 leader_election.cc:302] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f67d8ca88d4746afb89eeb62452ee1b3; no voters:
I20260925 05:05:05.721239 15863 leader_election.cc:288] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 05:05:05.721366 15867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 05:05:05.722576 15867 raft_consensus.cc:686] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [term 3 LEADER]: Becoming Leader. State: Replica: f67d8ca88d4746afb89eeb62452ee1b3, State: Running, Role: LEADER
I20260925 05:05:05.723173 15867 consensus_queue.cc:237] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d8ca88d4746afb89eeb62452ee1b3" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36613 } }
I20260925 05:05:05.723378 15863 sys_catalog.cc:564] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:05.725945 15869 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f67d8ca88d4746afb89eeb62452ee1b3. Latest consensus state: current_term: 3 leader_uuid: "f67d8ca88d4746afb89eeb62452ee1b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d8ca88d4746afb89eeb62452ee1b3" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36613 } } }
I20260925 05:05:05.726311 15869 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:05.726028 15868 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f67d8ca88d4746afb89eeb62452ee1b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d8ca88d4746afb89eeb62452ee1b3" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36613 } } }
I20260925 05:05:05.726430 15868 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:05.727190 15872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:05.742376 15872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:05.743953 15872 catalog_manager.cc:1259] Loaded cluster ID: b082fe482e694e25915b0a16eb40d143
I20260925 05:05:05.744115 15872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:05:05.748190 15872 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:05.749817 15872 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3: Loaded TSK: 0
I20260925 05:05:05.750031 15872 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f67d8ca88d4746afb89eeb62452ee1b3: Loaded TSK: 1
F20260925 05:05:05.751089 15872 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff5d64859ad google::LogMessage::Fail() at ??:0
@ 0x7ff5d648888a google::LogMessage::SendToLog() at ??:0
@ 0x7ff5d64854de google::LogMessage::Flush() at ??:0
@ 0x7ff5d6486a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff5da1e97d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff5da1d873e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff5ebeb41c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff5ebeb3783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff5ebf3f2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7ff5ebf3f0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7ff5ebf895e9 std::function<>::operator()() at ??:0
@ 0x7ff5ebeb7cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ff5ebeb5c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff5ebf3d610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ff5ebf3d439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff5d90d1345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff5d90d491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff5d90d4799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff5d90afeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff5dc3896db start_thread at ??:0
@ 0x7ff5d531f61f clone at ??:0
W20260925 05:05:06.092605 15865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:06.092880 15865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:06.092974 15865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:06.100032 15865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:06.100128 15865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:06.100237 15865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:05:06.109560 15865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:36567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=41357
--tserver_master_addrs=127.30.247.62:36613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:06.110680 15865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:06.111920 15865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:06.124512 15908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:06.124565 15909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:06.124619 15911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:06.125393 15865 server_base.cc:1034] running on GCE node
I20260925 05:05:06.125914 15865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:06.126715 15865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:06.127930 15865 hybrid_clock.cc:648] HybridClock initialized: now 1790312706127880 us; error 36 us; skew 500 ppm
I20260925 05:05:06.128194 15865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:05:06 dist-test-slave-5n86 krb5kdc[14718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312706, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:06.131316 15865 init.cc:377] Logged in from keytab as kudu/127.30.247.1@KRBTEST.COM (short username kudu)
I20260925 05:05:06.132668 15865 webserver.cc:460] Webserver started at http://127.30.247.1:41357/ using document root <none> and password file <none>
I20260925 05:05:06.133271 15865 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:06.133368 15865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:06.138382 15865 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 05:05:06.140987 15918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:06.142261 15865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:05:06.142434 15865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "2efd1e14d8494b48b714b28b1476fb52"
format_stamp: "Formatted at 2026-09-25 05:04:20 on dist-test-slave-5n86"
server_key: "d13c5f23da204ecd2873a9dab89c50e0"
server_key_iv: "aee8b583550b09d80e2b38b3585601fa"
server_key_version: "encryptionkey@0"
I20260925 05:05:06.143359 15865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:06.198457 15865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:06.202363 15865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:06.202641 15865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:06.203617 15865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:06.205098 15865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:06.205215 15865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:06.205343 15865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:06.205430 15865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:06.238768 15865 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:36567
I20260925 05:05:06.238854 16030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:36567 every 8 connection(s)
I20260925 05:05:06.240217 15865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:05:06.245139 16033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:06.243199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.1:46153)
0925 05:05:06.243924 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:06.244224 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":270,"threads_started":1}
W20260925 05:05:06.245779 16031 heartbeater.cc:644] Failed to heartbeat to 127.30.247.62:36613 (0 consecutive failures): Network error: Failed to ping master at 127.30.247.62:36613: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
I20260925 05:05:06.246517 16033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:06.246257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.1:46173)
0925 05:05:06.246357 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:06.246419 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 05:05:06.247052 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 15865
I20260925 05:05:06.247882 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:46345
--local_ip_for_outbound_sockets=127.30.247.2
--tserver_master_addrs=127.30.247.62:36613
--webserver_port=32893
--webserver_interface=127.30.247.2
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:05:06.248081 16033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:06.247725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.1:48627)
0925 05:05:06.247844 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:06.247922 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 05:05:06.248322 16031 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 05:05:06.565014 16034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:06.565260 16034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:06.565349 16034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:06.572141 16034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:06.572235 16034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:06.572340 16034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:05:06.581460 16034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:46345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=32893
--tserver_master_addrs=127.30.247.62:36613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:06.582621 16034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:06.583837 16034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:06.596357 16042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:06.596442 16040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:06.596627 16039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:06.597465 16034 server_base.cc:1034] running on GCE node
I20260925 05:05:06.598050 16034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:06.598923 16034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:06.600157 16034 hybrid_clock.cc:648] HybridClock initialized: now 1790312706600115 us; error 54 us; skew 500 ppm
I20260925 05:05:06.600468 16034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:05:06 dist-test-slave-5n86 krb5kdc[14718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312706, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:06.603475 16034 init.cc:377] Logged in from keytab as kudu/127.30.247.2@KRBTEST.COM (short username kudu)
I20260925 05:05:06.604779 16034 webserver.cc:460] Webserver started at http://127.30.247.2:32893/ using document root <none> and password file <none>
I20260925 05:05:06.605373 16034 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:06.605468 16034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:06.610316 16034 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 05:05:06.612949 16049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:06.614161 16034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:05:06.614282 16034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "46c2a23747a5451ab3e440a2b7549491"
format_stamp: "Formatted at 2026-09-25 05:04:21 on dist-test-slave-5n86"
server_key: "c6ac0cf3e4ac96cb8852b7eccd2e9ba9"
server_key_iv: "e334f34cc78c0d54a3e477c2ec9bf809"
server_key_version: "encryptionkey@0"
I20260925 05:05:06.615176 16034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:06.628013 16034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:06.631819 16034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:06.632032 16034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:06.632917 16034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:06.634097 16034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:06.634181 16034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:06.634246 16034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:06.634301 16034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:06.666674 16034 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:46345
I20260925 05:05:06.666749 16161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:46345 every 8 connection(s)
I20260925 05:05:06.668110 16034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:05:06.669627 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 16034
I20260925 05:05:06.670338 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:35489
--local_ip_for_outbound_sockets=127.30.247.3
--tserver_master_addrs=127.30.247.62:36613
--webserver_port=34513
--webserver_interface=127.30.247.3
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:05:06.673494 16164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:06.671194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.2:47795)
0925 05:05:06.672050 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:06.672349 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":323,"threads_started":1}
W20260925 05:05:06.674223 16162 heartbeater.cc:644] Failed to heartbeat to 127.30.247.62:36613 (0 consecutive failures): Network error: Failed to ping master at 127.30.247.62:36613: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
I20260925 05:05:06.675180 16164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:06.674736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.2:50329)
0925 05:05:06.674922 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:06.675021 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 05:05:06.676128 16164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:06.675727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.2:35523)
0925 05:05:06.675890 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:06.675983 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 05:05:06.676446 16162 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 05:05:06.993104 16165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:06.993546 16165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:06.993722 16165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:07.001101 16165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:07.001209 16165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:07.001380 16165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:05:07.011248 16165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:35489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=34513
--tserver_master_addrs=127.30.247.62:36613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:07.012681 16165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:07.014019 16165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:07.028005 16173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:07.027978 16171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:07.027923 16170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:07.029209 16165 server_base.cc:1034] running on GCE node
I20260925 05:05:07.029830 16165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:07.030807 16165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:07.032048 16165 hybrid_clock.cc:648] HybridClock initialized: now 1790312707031996 us; error 47 us; skew 500 ppm
I20260925 05:05:07.032380 16165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:05:07 dist-test-slave-5n86 krb5kdc[14718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312707, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:07.035485 16165 init.cc:377] Logged in from keytab as kudu/127.30.247.3@KRBTEST.COM (short username kudu)
I20260925 05:05:07.037039 16165 webserver.cc:460] Webserver started at http://127.30.247.3:34513/ using document root <none> and password file <none>
I20260925 05:05:07.037763 16165 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:07.037937 16165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:07.043759 16165 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 05:05:07.046782 16180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:07.048210 16165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:05:07.048388 16165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "501237da7c2c4854b627f5015f66fb70"
format_stamp: "Formatted at 2026-09-25 05:04:21 on dist-test-slave-5n86"
server_key: "e19eb6e09693392c367bc2b032278067"
server_key_iv: "4470128858c23210cbf297db32f12ee7"
server_key_version: "encryptionkey@0"
I20260925 05:05:07.049384 16165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:07.064596 16165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:07.068845 16165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:07.069128 16165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:07.070097 16165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:07.071425 16165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:07.071519 16165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:07.071677 16165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:07.071780 16165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:07.105015 16165 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:35489
I20260925 05:05:07.105132 16292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:35489 every 8 connection(s)
I20260925 05:05:07.106469 16165 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:05:07.111331 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:07.109496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.3:37365)
0925 05:05:07.110246 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:07.110485 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":282,"threads_started":1}
W20260925 05:05:07.111996 16293 heartbeater.cc:644] Failed to heartbeat to 127.30.247.62:36613 (0 consecutive failures): Network error: Failed to ping master at 127.30.247.62:36613: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
I20260925 05:05:07.112960 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:07.112614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.3:43219)
0925 05:05:07.112765 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:07.112849 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 05:05:07.113788 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:07.113454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.3:39313)
0925 05:05:07.113604 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:07.113688 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 05:05:07.114018 16293 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 05:05:07.114768 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 16165
I20260925 05:05:07.249790 16297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:07.248915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.1:48229)
0925 05:05:07.249410 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:07.249544 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":260,"threads_started":1}
I20260925 05:05:07.677711 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:07.676953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.2:42621)
0925 05:05:07.677394 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:07.677495 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":216,"threads_started":1}
I20260925 05:05:08.115247 16299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:08.114505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.3:56637)
0925 05:05:08.114928 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:08.115029 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":215,"threads_started":1}
I20260925 05:05:08.251291 16300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:08.250637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.1:60971)
0925 05:05:08.251057 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:08.251132 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":225,"threads_started":1}
I20260925 05:05:08.679301 16301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:08.678614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.2:35823)
0925 05:05:08.679067 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:08.679143 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":234,"threads_started":1}
I20260925 05:05:09.116883 16302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:09.116124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.3:37243)
0925 05:05:09.116639 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:09.116720 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":225,"threads_started":1}
I20260925 05:05:09.252993 16303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:09.252189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.1:43131)
0925 05:05:09.252722 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:09.252803 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":278,"threads_started":1}
I20260925 05:05:09.680879 16304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:09.680123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.2:53367)
0925 05:05:09.680615 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:09.680698 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":224,"threads_started":1}
I20260925 05:05:10.118661 16305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:10.117863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.3:38437)
0925 05:05:10.118313 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:10.118404 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":237,"threads_started":1}
I20260925 05:05:10.254575 16306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:10.253879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.1:42875)
0925 05:05:10.254330 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:10.254417 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":208,"threads_started":1}
I20260925 05:05:10.682423 16307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:10.681706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.2:47711)
0925 05:05:10.682150 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:10.682230 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":215,"threads_started":1}
I20260925 05:05:11.120532 16308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:11.119669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.3:39793)
0925 05:05:11.120168 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:11.120255 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":241,"threads_started":1}
I20260925 05:05:11.256132 16309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:11.255394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.1:44063)
0925 05:05:11.255861 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:11.255972 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":245,"threads_started":1}
I20260925 05:05:11.684171 16310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:11.683446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.2:35935)
0925 05:05:11.683901 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:11.683989 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":239,"threads_started":1}
I20260925 05:05:12.122103 16311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:12.121419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.3:48971)
0925 05:05:12.121871 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:12.121944 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":241,"threads_started":1}
I20260925 05:05:12.257715 16312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:12.257043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.1:46647)
0925 05:05:12.257458 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:12.257546 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":222,"threads_started":1}
I20260925 05:05:12.685761 16313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:12.685069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.2:37947)
0925 05:05:12.685521 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:12.685608 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":215,"threads_started":1}
I20260925 05:05:13.123703 16314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:13.123032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.3:55777)
0925 05:05:13.123451 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:13.123541 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":216,"threads_started":1}
I20260925 05:05:13.259281 16315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:13.258532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.1:58667)
0925 05:05:13.258978 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:13.259075 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":203,"threads_started":1}
I20260925 05:05:13.687441 16316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:13.686699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.2:53145)
0925 05:05:13.687156 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:13.687234 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":206,"threads_started":1}
I20260925 05:05:14.125439 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:14.124678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.3:45969)
0925 05:05:14.125145 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:14.125225 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":206,"threads_started":1}
I20260925 05:05:14.260808 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:14.260114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.1:50455)
0925 05:05:14.260511 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:14.260651 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":210,"threads_started":1}
I20260925 05:05:14.689034 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:14.688252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.2:57259)
0925 05:05:14.688769 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:14.688858 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":289,"threads_started":1}
I20260925 05:05:15.126986 16320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:15.126328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.3:51635)
0925 05:05:15.126755 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:15.126830 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":232,"threads_started":1}
I20260925 05:05:15.262521 16321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:15.261764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.1:33781)
0925 05:05:15.262207 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:15.262286 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":215,"threads_started":1}
I20260925 05:05:15.690675 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:15.689917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.2:38115)
0925 05:05:15.690410 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:15.690500 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":246,"threads_started":1}
I20260925 05:05:16.128649 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:16.127874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.3:41097)
0925 05:05:16.128328 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:16.128404 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":216,"threads_started":1}
I20260925 05:05:16.264253 16324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:16.263526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.1:59755)
0925 05:05:16.264004 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:16.264086 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":265,"threads_started":1}
I20260925 05:05:16.692387 16325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:16.691691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.2:52015)
0925 05:05:16.692148 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:16.692226 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":255,"threads_started":1}
I20260925 05:05:17.130319 16326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:17.129530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.3:33157)
0925 05:05:17.130082 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:17.130169 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":331,"threads_started":1}
I20260925 05:05:17.265849 16327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:17.265156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.1:57167)
0925 05:05:17.265586 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:17.265665 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":223,"threads_started":1}
I20260925 05:05:17.694099 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:17.693331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.2:43877)
0925 05:05:17.693818 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:17.693910 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":249,"threads_started":1}
I20260925 05:05:18.131965 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:18.131244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.3:57341)
0925 05:05:18.131702 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:18.131791 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":243,"threads_started":1}
I20260925 05:05:18.267442 16330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:18.266719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.1:32863)
0925 05:05:18.267187 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:18.267274 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":233,"threads_started":1}
I20260925 05:05:18.695714 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:18.694971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.2:45403)
0925 05:05:18.695478 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:18.695556 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":287,"threads_started":1}
I20260925 05:05:19.133579 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:19.132889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.3:40917)
0925 05:05:19.133330 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:19.133408 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":235,"threads_started":1}
I20260925 05:05:19.269188 16333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:19.268384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.1:47999)
0925 05:05:19.268917 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:19.269007 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":318,"threads_started":1}
I20260925 05:05:19.697409 16334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:19.696677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.2:43427)
0925 05:05:19.697136 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:19.697228 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":213,"threads_started":1}
I20260925 05:05:20.135231 16335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:20.134479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.3:49835)
0925 05:05:20.134932 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:20.135040 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":241,"threads_started":1}
I20260925 05:05:20.270754 16336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:20.270042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.1:47337)
0925 05:05:20.270476 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:20.270574 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":224,"threads_started":1}
I20260925 05:05:20.699046 16337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:20.698372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.2:38137)
0925 05:05:20.698778 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:20.698857 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":200,"threads_started":1}
I20260925 05:05:21.136826 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:21.136063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.3:59469)
0925 05:05:21.136487 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:21.136641 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":229,"threads_started":1}
I20260925 05:05:21.272581 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:21.271747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.1:58845)
0925 05:05:21.272182 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:21.272307 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":228,"threads_started":1}
I20260925 05:05:21.700836 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:21.700108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.2:53465)
0925 05:05:21.700515 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:21.700648 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":216,"threads_started":1}
I20260925 05:05:22.122355 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 15865
I20260925 05:05:22.138579 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:22.137835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36613 (local address 127.30.247.3:37597)
0925 05:05:22.138295 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:22.138387 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":258,"threads_started":1}
I20260925 05:05:22.155725 16341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:22.123412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35176 (local address 127.30.247.1:36567)
0925 05:05:22.123989 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:22.124008 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:22.124058 (+ 50us) server_negotiation.cc:407] Connection header received
0925 05:05:22.125629 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:22.125638 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:22.126186 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:22.126483 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:22.127786 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:22.129033 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:22.130269 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:22.130899 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:22.131028 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:22.149498 (+ 18470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:22.149508 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:22.149524 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:22.149567 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:22.152689 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:22.153372 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:22.153378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:22.153384 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:22.153537 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:22.153932 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:22.153936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:22.153938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:22.154435 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:22.154652 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:22.155027 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 05:05:22.155445 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":292,"threads_started":1}
I20260925 05:05:22.157130 16026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:22.240463 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 15865
I20260925 05:05:22.257764 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 16034
I20260925 05:05:22.275455 16344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:22.258837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50458 (local address 127.30.247.2:46345)
0925 05:05:22.259360 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:22.259374 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:22.259412 (+ 38us) server_negotiation.cc:407] Connection header received
0925 05:05:22.261072 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:22.261082 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:22.261524 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:22.261882 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:22.263362 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:22.264647 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:22.265899 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:22.266583 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:22.266797 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:22.269412 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:22.269421 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:22.269435 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:22.269476 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:22.272184 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:22.272924 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:22.272931 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:22.272937 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:22.273098 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:22.273478 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:22.273484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:22.273487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:22.273982 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:22.274323 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:22.274750 (+ 427us) server_negotiation.cc:299] Negotiation successful
0925 05:05:22.275166 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":251,"threads_started":1}
I20260925 05:05:22.276927 16157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:22.362396 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 16034
I20260925 05:05:22.379102 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 16165
I20260925 05:05:22.396087 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:22.380140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52834 (local address 127.30.247.3:35489)
0925 05:05:22.380760 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:22.380773 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:22.380813 (+ 40us) server_negotiation.cc:407] Connection header received
0925 05:05:22.382432 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:22.382442 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:22.382857 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:22.383131 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:22.384429 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:22.385706 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:22.386924 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:22.387540 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:22.387679 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:22.390332 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:22.390340 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:22.390353 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:22.390385 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:22.393237 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:22.393882 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:22.393888 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:22.393893 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:22.394040 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:22.394399 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:22.394402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:22.394404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:22.394849 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:22.395055 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:22.395405 (+ 350us) server_negotiation.cc:299] Negotiation successful
0925 05:05:22.395806 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":289,"threads_started":1}
I20260925 05:05:22.397509 16288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:22.480226 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 16165
2026-09-25T05:05:22Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64156 ms)
[----------] 30 tests from SecurityITest (291966 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:05:22 dist-test-slave-5n86 krb5kdc[16350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:05:22 dist-test-slave-5n86 krb5kdc[16350](info): set up 2 sockets
Sep 25 05:05:22 dist-test-slave-5n86 krb5kdc[16350](info): commencing operation
krb5kdc: starting...
W20260925 05:05:23.636960 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:05:23 dist-test-slave-5n86 krb5kdc[16350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:05:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:05:23.868950 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:37371
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:46841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:37371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:24.187060 16363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:24.187387 16363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:24.187546 16363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:24.194753 16363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:24.194892 16363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:24.194998 16363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:24.195087 16363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:24.195165 16363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:24.205050 16363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:46841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:37371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:37371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:24.206326 16363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:24.207690 16363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:24.219676 16368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:24.219707 16369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:24.219745 16371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:24.220237 16363 server_base.cc:1034] running on GCE node
I20260925 05:05:24.221071 16363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:24.222397 16363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:24.223620 16363 hybrid_clock.cc:648] HybridClock initialized: now 1790312724223573 us; error 39 us; skew 500 ppm
I20260925 05:05:24.223959 16363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:05:24 dist-test-slave-5n86 krb5kdc[16350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312724, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:24.226929 16363 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:05:24.228318 16363 webserver.cc:460] Webserver started at http://127.30.247.62:35797/ using document root <none> and password file <none>
I20260925 05:05:24.229031 16363 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:24.229171 16363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:24.229468 16363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:24.232450 16363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "0d7da84a387447f2824b44ca68e57ab4"
format_stamp: "Formatted at 2026-09-25 05:05:24 on dist-test-slave-5n86"
server_key: "b03e612d62000fae6d4cf4c3a72cd0ca"
server_key_iv: "43a2d455fdb16519a0e5102e776cd98c"
server_key_version: "encryptionkey@0"
I20260925 05:05:24.233227 16363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "0d7da84a387447f2824b44ca68e57ab4"
format_stamp: "Formatted at 2026-09-25 05:05:24 on dist-test-slave-5n86"
server_key: "b03e612d62000fae6d4cf4c3a72cd0ca"
server_key_iv: "43a2d455fdb16519a0e5102e776cd98c"
server_key_version: "encryptionkey@0"
I20260925 05:05:24.238168 16363 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 05:05:24.241592 16378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:24.242877 16363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:05:24.243078 16363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "0d7da84a387447f2824b44ca68e57ab4"
format_stamp: "Formatted at 2026-09-25 05:05:24 on dist-test-slave-5n86"
server_key: "b03e612d62000fae6d4cf4c3a72cd0ca"
server_key_iv: "43a2d455fdb16519a0e5102e776cd98c"
server_key_version: "encryptionkey@0"
I20260925 05:05:24.243265 16363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:24.266547 16363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:24.270550 16363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:24.270830 16363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:24.289166 16363 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:37371
I20260925 05:05:24.289201 16429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:37371 every 8 connection(s)
I20260925 05:05:24.290500 16363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:24.294096 16430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:24.300881 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 16363
I20260925 05:05:24.301110 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:05:24.301923 31708 external_mini_cluster.cc:1428] Setting key 9a144b07482a25844766dee98d06fae0
I20260925 05:05:24.305174 16430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d7da84a387447f2824b44ca68e57ab4: Bootstrap starting.
I20260925 05:05:24.308709 16430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d7da84a387447f2824b44ca68e57ab4: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:24.309868 16430 log.cc:824] T 00000000000000000000000000000000 P 0d7da84a387447f2824b44ca68e57ab4: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:24.312625 16430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d7da84a387447f2824b44ca68e57ab4: No bootstrap required, opened a new log
Sep 25 05:05:24 dist-test-slave-5n86 krb5kdc[16350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:05:24.319988 16430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d7da84a387447f2824b44ca68e57ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d7da84a387447f2824b44ca68e57ab4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37371 } }
I20260925 05:05:24.320358 16430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d7da84a387447f2824b44ca68e57ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:24.320477 16430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d7da84a387447f2824b44ca68e57ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7da84a387447f2824b44ca68e57ab4, State: Initialized, Role: FOLLOWER
I20260925 05:05:24.321049 16430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d7da84a387447f2824b44ca68e57ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d7da84a387447f2824b44ca68e57ab4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37371 } }
I20260925 05:05:24.321442 16430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d7da84a387447f2824b44ca68e57ab4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:24.321590 16430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d7da84a387447f2824b44ca68e57ab4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:24.321777 16430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d7da84a387447f2824b44ca68e57ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:24.324633 16433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:24.303985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36172 (local address 127.30.247.62:37371)
0925 05:05:24.304759 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:24.304772 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:24.304812 (+ 40us) server_negotiation.cc:407] Connection header received
0925 05:05:24.306602 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:24.306624 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:24.307015 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:24.307327 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:24.309190 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:24.310585 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:24.311926 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:24.312642 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:24.312832 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:24.316907 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:24.316917 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:24.316931 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:24.316971 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:24.320103 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:24.321111 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:24.321118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:24.321124 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:24.321347 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:24.321818 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:24.321821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:24.321823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:24.322402 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:24.322730 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:24.323216 (+ 486us) server_negotiation.cc:299] Negotiation successful
0925 05:05:24.323611 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":355,"threads_started":1}
I20260925 05:05:24.324775 16430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d7da84a387447f2824b44ca68e57ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d7da84a387447f2824b44ca68e57ab4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37371 } }
I20260925 05:05:24.325313 16430 leader_election.cc:302] T 00000000000000000000000000000000 P 0d7da84a387447f2824b44ca68e57ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d7da84a387447f2824b44ca68e57ab4; no voters:
I20260925 05:05:24.325892 16430 leader_election.cc:288] T 00000000000000000000000000000000 P 0d7da84a387447f2824b44ca68e57ab4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:24.326051 16435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d7da84a387447f2824b44ca68e57ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:24.327840 16430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d7da84a387447f2824b44ca68e57ab4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:24.328617 16435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d7da84a387447f2824b44ca68e57ab4 [term 1 LEADER]: Becoming Leader. State: Replica: 0d7da84a387447f2824b44ca68e57ab4, State: Running, Role: LEADER
I20260925 05:05:24.329128 16435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d7da84a387447f2824b44ca68e57ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d7da84a387447f2824b44ca68e57ab4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37371 } }
I20260925 05:05:24.332135 16436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d7da84a387447f2824b44ca68e57ab4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d7da84a387447f2824b44ca68e57ab4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d7da84a387447f2824b44ca68e57ab4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37371 } } }
I20260925 05:05:24.332765 16436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d7da84a387447f2824b44ca68e57ab4 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:24.333356 16437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d7da84a387447f2824b44ca68e57ab4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d7da84a387447f2824b44ca68e57ab4. Latest consensus state: current_term: 1 leader_uuid: "0d7da84a387447f2824b44ca68e57ab4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d7da84a387447f2824b44ca68e57ab4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37371 } } }
I20260925 05:05:24.333730 16437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d7da84a387447f2824b44ca68e57ab4 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:24.334131 16443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:24.338593 16443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:24.347306 16443 catalog_manager.cc:1347] Generated new cluster ID: 2c09933fd74a4e188522bd09f816b250
I20260925 05:05:24.347409 16443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:05:24.367175 16443 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:24.368475 16443 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:24.382695 16443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d7da84a387447f2824b44ca68e57ab4: Generated new TSK 0
I20260925 05:05:24.383574 16443 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
I20260925 05:05:24.440641 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:0
--local_ip_for_outbound_sockets=127.30.247.1
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:37371
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:46841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:24.765944 16456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:24.766232 16456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:24.766390 16456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:24.773762 16456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:24.773880 16456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:24.774068 16456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:05:24.783871 16456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:46841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:37371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:24.785192 16456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:24.786515 16456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:24.799434 16462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:24.799382 16464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:24.799482 16461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:24.800423 16456 server_base.cc:1034] running on GCE node
I20260925 05:05:24.800997 16456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:24.801811 16456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:24.803030 16456 hybrid_clock.cc:648] HybridClock initialized: now 1790312724802994 us; error 49 us; skew 500 ppm
I20260925 05:05:24.803328 16456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:05:24 dist-test-slave-5n86 krb5kdc[16350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312724, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:24.806399 16456 init.cc:377] Logged in from keytab as kudu/127.30.247.1@KRBTEST.COM (short username kudu)
I20260925 05:05:24.807809 16456 webserver.cc:460] Webserver started at http://127.30.247.1:41087/ using document root <none> and password file <none>
I20260925 05:05:24.808437 16456 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:24.808589 16456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:24.808916 16456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:24.811646 16456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-0/data/instance:
uuid: "f2a5d065a7b349f283a68dab5a30bbff"
format_stamp: "Formatted at 2026-09-25 05:05:24 on dist-test-slave-5n86"
server_key: "198c0beeaed5c1424216227937d5c1ce"
server_key_iv: "4080bf7fba637fcc6acdf1a55f12d6a9"
server_key_version: "encryptionkey@0"
I20260925 05:05:24.812381 16456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance:
uuid: "f2a5d065a7b349f283a68dab5a30bbff"
format_stamp: "Formatted at 2026-09-25 05:05:24 on dist-test-slave-5n86"
server_key: "198c0beeaed5c1424216227937d5c1ce"
server_key_iv: "4080bf7fba637fcc6acdf1a55f12d6a9"
server_key_version: "encryptionkey@0"
I20260925 05:05:24.817359 16456 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 05:05:24.820595 16471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:24.821861 16456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:05:24.822026 16456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "f2a5d065a7b349f283a68dab5a30bbff"
format_stamp: "Formatted at 2026-09-25 05:05:24 on dist-test-slave-5n86"
server_key: "198c0beeaed5c1424216227937d5c1ce"
server_key_iv: "4080bf7fba637fcc6acdf1a55f12d6a9"
server_key_version: "encryptionkey@0"
I20260925 05:05:24.822217 16456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:24.851558 16456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:24.855702 16456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:24.855984 16456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:24.856927 16456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:24.858187 16456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:24.858285 16456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:24.858419 16456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:24.858497 16456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:24.891466 16456 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:46855
I20260925 05:05:24.891546 16583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:46855 every 8 connection(s)
I20260925 05:05:24.892814 16456 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:05:24.895272 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 16456
I20260925 05:05:24.895509 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance
I20260925 05:05:24.896252 31708 external_mini_cluster.cc:1428] Setting key 33a621c484ffeb68683c08531dffebe4
Sep 25 05:05:24 dist-test-slave-5n86 krb5kdc[16350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312724, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:05:24.914476 16586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:24.895285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:60249 (local address 127.30.247.62:37371)
0925 05:05:24.895804 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:24.895812 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:24.896875 (+ 1063us) server_negotiation.cc:407] Connection header received
0925 05:05:24.898680 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:24.898686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:24.898776 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:24.898895 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:24.900672 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:24.901812 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:24.903987 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:24.904734 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:24.904885 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:24.908854 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:24.908859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:24.908863 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:24.908893 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:24.911306 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:24.912226 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:24.912230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:24.912232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:24.912347 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:24.912862 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:24.912865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:24.912867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:24.913118 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:24.913250 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:24.914061 (+ 811us) server_negotiation.cc:299] Negotiation successful
0925 05:05:24.914231 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":243,"threads_started":1}
I20260925 05:05:24.915910 16587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:24.895870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37371 (local address 127.30.247.1:60249)
0925 05:05:24.896701 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:24.896761 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:24.898426 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:24.899121 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:24.899130 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:24.899768 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:24.900456 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:24.900479 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:24.901976 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:24.901983 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:24.903831 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:24.903848 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:24.904943 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:24.904951 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:24.905237 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:24.906029 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:24.906051 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:24.908636 (+ 2585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:24.911445 (+ 2809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:24.911456 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:24.911474 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:24.912099 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:24.912460 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:24.912468 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:24.912475 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:24.912752 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:24.913244 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:24.913254 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:24.913682 (+ 428us) client_negotiation.cc:769] Sending connection context
0925 05:05:24.913972 (+ 290us) client_negotiation.cc:240] Negotiation successful
0925 05:05:24.914492 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":336,"threads_started":1}
I20260925 05:05:24.917469 16584 heartbeater.cc:347] Connected to a master server at 127.30.247.62:37371
I20260925 05:05:24.917796 16584 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:24.918721 16584 heartbeater.cc:511] Master 127.30.247.62:37371 requested a full tablet report, sending...
I20260925 05:05:24.920848 16395 ts_manager.cc:194] Registered new tserver with Master: f2a5d065a7b349f283a68dab5a30bbff (127.30.247.1:46855)
WARNING: no policy specified for kudu/127.30.247.2@KRBTEST.COM; defaulting to no policy
I20260925 05:05:24.922386 16395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.1@KRBTEST.COM'} at 127.30.247.1:60249
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
I20260925 05:05:24.952034 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:0
--local_ip_for_outbound_sockets=127.30.247.2
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:37371
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:46841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:25.266350 16590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:25.266599 16590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:25.266687 16590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:25.273597 16590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:25.273694 16590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:25.273799 16590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:05:25.283025 16590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:46841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:37371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:25.284170 16590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:25.285450 16590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:25.299057 16595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:25.299136 16596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:25.299160 16598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:25.299866 16590 server_base.cc:1034] running on GCE node
I20260925 05:05:25.300496 16590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:25.301590 16590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:25.302824 16590 hybrid_clock.cc:648] HybridClock initialized: now 1790312725302764 us; error 50 us; skew 500 ppm
I20260925 05:05:25.303171 16590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:05:25 dist-test-slave-5n86 krb5kdc[16350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312725, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:25.306854 16590 init.cc:377] Logged in from keytab as kudu/127.30.247.2@KRBTEST.COM (short username kudu)
I20260925 05:05:25.309154 16590 webserver.cc:460] Webserver started at http://127.30.247.2:41199/ using document root <none> and password file <none>
I20260925 05:05:25.309880 16590 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:25.310024 16590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:25.310340 16590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:25.313227 16590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-1/data/instance:
uuid: "1494216f88f04fc7bf0dd96b46f7f7a5"
format_stamp: "Formatted at 2026-09-25 05:05:25 on dist-test-slave-5n86"
server_key: "6aa65c0b60ac2bfa28dd9a87342a1028"
server_key_iv: "03a44afa6bda160eda7a20580cc282b9"
server_key_version: "encryptionkey@0"
I20260925 05:05:25.313961 16590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance:
uuid: "1494216f88f04fc7bf0dd96b46f7f7a5"
format_stamp: "Formatted at 2026-09-25 05:05:25 on dist-test-slave-5n86"
server_key: "6aa65c0b60ac2bfa28dd9a87342a1028"
server_key_iv: "03a44afa6bda160eda7a20580cc282b9"
server_key_version: "encryptionkey@0"
I20260925 05:05:25.318863 16590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:05:25.322546 16605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:25.323760 16590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:05:25.323938 16590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "1494216f88f04fc7bf0dd96b46f7f7a5"
format_stamp: "Formatted at 2026-09-25 05:05:25 on dist-test-slave-5n86"
server_key: "6aa65c0b60ac2bfa28dd9a87342a1028"
server_key_iv: "03a44afa6bda160eda7a20580cc282b9"
server_key_version: "encryptionkey@0"
I20260925 05:05:25.324126 16590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:25.345410 16590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:25.349637 16590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:25.349931 16590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:25.350893 16590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:25.352229 16590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:25.352340 16590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:25.352492 16590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:25.352588 16590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260925 05:05:25.385890 16590 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:37855
I20260925 05:05:25.385941 16717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:37855 every 8 connection(s)
I20260925 05:05:25.387204 16590 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:05:25.391912 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 16590
I20260925 05:05:25.392091 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance
I20260925 05:05:25.392578 31708 external_mini_cluster.cc:1428] Setting key 408c76214a8601d002f7b0ad1e003a02
Sep 25 05:05:25 dist-test-slave-5n86 krb5kdc[16350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312725, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:05:25.408066 16586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:25.389789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:58881 (local address 127.30.247.62:37371)
0925 05:05:25.389989 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:25.389994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:25.391143 (+ 1149us) server_negotiation.cc:407] Connection header received
0925 05:05:25.393110 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:25.393121 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:25.393196 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:25.393289 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:25.395156 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:25.396283 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:25.397906 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:25.398651 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:25.398793 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:25.402036 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:25.402045 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:25.402053 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:25.402098 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:25.404865 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:25.405759 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:25.405763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:25.405765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:25.405887 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:25.406354 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:25.406357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:25.406359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:25.406629 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:25.406748 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:25.407633 (+ 885us) server_negotiation.cc:299] Negotiation successful
0925 05:05:25.407809 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 05:05:25.409497 16720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:25.390227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37371 (local address 127.30.247.2:58881)
0925 05:05:25.390968 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:25.391016 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:25.392849 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:25.393581 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:25.393601 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:25.394313 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:25.394981 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:25.394997 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:25.396450 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:25.396454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:25.397753 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:25.397764 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:25.398848 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:25.398853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:25.399040 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:25.399687 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:25.399709 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:25.401816 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:25.404990 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:25.405001 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:25.405019 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:25.405644 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:25.405991 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:25.405998 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:25.406004 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:25.406234 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:25.406761 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:25.406772 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:25.407207 (+ 435us) client_negotiation.cc:769] Sending connection context
0925 05:05:25.407494 (+ 287us) client_negotiation.cc:240] Negotiation successful
0925 05:05:25.408060 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":310,"threads_started":1}
I20260925 05:05:25.410905 16718 heartbeater.cc:347] Connected to a master server at 127.30.247.62:37371
I20260925 05:05:25.411193 16718 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:25.411895 16718 heartbeater.cc:511] Master 127.30.247.62:37371 requested a full tablet report, sending...
I20260925 05:05:25.413296 16395 ts_manager.cc:194] Registered new tserver with Master: 1494216f88f04fc7bf0dd96b46f7f7a5 (127.30.247.2:37855)
I20260925 05:05:25.414072 16395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.2@KRBTEST.COM'} at 127.30.247.2:58881
WARNING: no policy specified for kudu/127.30.247.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
I20260925 05:05:25.448174 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:0
--local_ip_for_outbound_sockets=127.30.247.3
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:37371
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:46841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:25.766837 16723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:25.767180 16723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:25.767360 16723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:25.775190 16723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:25.775337 16723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:25.775529 16723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:05:25.785473 16723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:46841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:37371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:25.786710 16723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:25.788034 16723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:25.800800 16729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:25.800840 16731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:25.800762 16728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:25.801476 16723 server_base.cc:1034] running on GCE node
I20260925 05:05:25.802096 16723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:25.802980 16723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:25.804201 16723 hybrid_clock.cc:648] HybridClock initialized: now 1790312725804149 us; error 49 us; skew 500 ppm
I20260925 05:05:25.804562 16723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:05:25 dist-test-slave-5n86 krb5kdc[16350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312725, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:25.807612 16723 init.cc:377] Logged in from keytab as kudu/127.30.247.3@KRBTEST.COM (short username kudu)
I20260925 05:05:25.808957 16723 webserver.cc:460] Webserver started at http://127.30.247.3:33461/ using document root <none> and password file <none>
I20260925 05:05:25.809607 16723 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:25.809743 16723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:25.810037 16723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:25.812855 16723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-2/data/instance:
uuid: "1a15398be78d45d2881b81baf4eda359"
format_stamp: "Formatted at 2026-09-25 05:05:25 on dist-test-slave-5n86"
server_key: "5b87a1207138ccee692bb47637ebf21c"
server_key_iv: "ef176af6d9b60f272da6f25385c3571f"
server_key_version: "encryptionkey@0"
I20260925 05:05:25.813563 16723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance:
uuid: "1a15398be78d45d2881b81baf4eda359"
format_stamp: "Formatted at 2026-09-25 05:05:25 on dist-test-slave-5n86"
server_key: "5b87a1207138ccee692bb47637ebf21c"
server_key_iv: "ef176af6d9b60f272da6f25385c3571f"
server_key_version: "encryptionkey@0"
I20260925 05:05:25.818454 16723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:05:25.821707 16738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:25.822960 16723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 05:05:25.823081 16723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "1a15398be78d45d2881b81baf4eda359"
format_stamp: "Formatted at 2026-09-25 05:05:25 on dist-test-slave-5n86"
server_key: "5b87a1207138ccee692bb47637ebf21c"
server_key_iv: "ef176af6d9b60f272da6f25385c3571f"
server_key_version: "encryptionkey@0"
I20260925 05:05:25.823232 16723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:25.838989 16723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:25.842602 16723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:25.842820 16723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:25.843626 16723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:25.844825 16723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:25.844893 16723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:25.844980 16723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:25.845022 16723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:25.878167 16723 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:45645
I20260925 05:05:25.878213 16850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:45645 every 8 connection(s)
I20260925 05:05:25.879469 16723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:05:25.888916 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 16723
I20260925 05:05:25.889122 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance
I20260925 05:05:25.889716 31708 external_mini_cluster.cc:1428] Setting key 71ad8b0a5b12e6c443019e5c1dc1d836
Sep 25 05:05:25 dist-test-slave-5n86 krb5kdc[16350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312725, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:05:25.899104 16586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:25.882144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:56153 (local address 127.30.247.62:37371)
0925 05:05:25.882327 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:25.882332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:25.883420 (+ 1088us) server_negotiation.cc:407] Connection header received
0925 05:05:25.885253 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:25.885258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:25.885333 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:25.885432 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:25.886884 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:25.887725 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:25.889369 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:25.890017 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:25.890153 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:25.894213 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:25.894218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:25.894221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:25.894252 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:25.896716 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:25.897332 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:25.897336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:25.897338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:25.897449 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:25.897796 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:25.897798 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:25.897800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:25.898049 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:25.898153 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:25.898710 (+ 557us) server_negotiation.cc:299] Negotiation successful
0925 05:05:25.898872 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:05:25.899933 16853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:25.882567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:37371 (local address 127.30.247.3:56153)
0925 05:05:25.883262 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:25.883302 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:25.884969 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:25.885648 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:25.885655 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:25.886165 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:25.886713 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:25.886730 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:25.887860 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:25.887863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:25.889220 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:25.889236 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:25.890219 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:25.890224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:25.890535 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:25.891306 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:25.891338 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:25.894030 (+ 2692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:25.896836 (+ 2806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:25.896841 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:25.896852 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:25.897231 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:25.897533 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:25.897536 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:25.897538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:25.897700 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:25.898143 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:25.898148 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:25.898415 (+ 267us) client_negotiation.cc:769] Sending connection context
0925 05:05:25.898619 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 05:05:25.898963 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":283,"threads_started":1}
I20260925 05:05:25.901266 16851 heartbeater.cc:347] Connected to a master server at 127.30.247.62:37371
I20260925 05:05:25.901503 16851 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:25.902107 16851 heartbeater.cc:511] Master 127.30.247.62:37371 requested a full tablet report, sending...
I20260925 05:05:25.903368 16395 ts_manager.cc:194] Registered new tserver with Master: 1a15398be78d45d2881b81baf4eda359 (127.30.247.3:45645)
I20260925 05:05:25.904311 16395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.3@KRBTEST.COM'} at 127.30.247.3:56153
I20260925 05:05:25.905210 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:05:25.922178 16586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:25.909651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36178 (local address 127.30.247.62:37371)
0925 05:05:25.909863 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:25.909872 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:25.910215 (+ 343us) server_negotiation.cc:407] Connection header received
0925 05:05:25.910406 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:25.910411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:25.910475 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:25.910570 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:25.911978 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:25.912740 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:25.914016 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:25.914428 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:25.914562 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:25.917128 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:25.917134 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:25.917137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:25.917173 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:25.919472 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:25.920156 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:25.920161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:25.920170 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:25.920293 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:25.920756 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:25.920766 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:25.920768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:25.921061 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:25.921338 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:25.921636 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 05:05:25.921861 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 05:05:25.925343 16584 heartbeater.cc:503] Master 127.30.247.62:37371 was elected leader, sending a full tablet report...
I20260925 05:05:25.926324 16394 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:05:25.933784 16394 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:05:25.970266 16865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:25.945669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41670 (local address 127.30.247.3:45645)
0925 05:05:25.958289 (+ 12620us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:25.958301 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:25.959087 (+ 786us) server_negotiation.cc:407] Connection header received
0925 05:05:25.959206 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:25.959214 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:25.959484 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:25.959707 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:25.961031 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:25.962933 (+ 1902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:25.967552 (+ 4619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:25.969034 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:25.969166 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:25.969403 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:25.969738 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 05:05:25.969910 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12430,"thread_start_us":12271,"threads_started":1}
I20260925 05:05:25.970285 16864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:25.945417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:45645 (local address 127.0.0.1:41670)
0925 05:05:25.958411 (+ 12994us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:25.958469 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:25.958690 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:25.959990 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:25.959994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:25.960018 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:25.960699 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:25.960715 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:25.963147 (+ 2432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:25.963154 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:25.966458 (+ 3304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:25.966479 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:25.969252 (+ 2773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:25.969258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:25.969462 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:25.969527 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 05:05:25.969651 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 05:05:25.969848 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12688,"mutex_wait_us":144,"spinlock_wait_cycles":2816,"thread_start_us":14302,"threads_started":1}
I20260925 05:05:25.971393 16863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:25.945689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:46855 (local address 127.0.0.1:53046)
0925 05:05:25.958771 (+ 13082us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:25.958819 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:25.958992 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:25.959896 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:25.959906 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:25.959939 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:25.960980 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:25.960993 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:25.963170 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:25.963178 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:25.968425 (+ 5247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:25.968437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:25.970503 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:25.970508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:25.970671 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:25.970706 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 05:05:25.970785 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 05:05:25.970913 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12904,"mutex_wait_us":2174,"thread_start_us":13206,"threads_started":1}
I20260925 05:05:25.972218 16866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:25.945985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53046 (local address 127.30.247.1:46855)
0925 05:05:25.958766 (+ 12781us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:25.958774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:25.959010 (+ 236us) server_negotiation.cc:407] Connection header received
0925 05:05:25.959276 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:25.959283 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:25.959553 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:25.960527 (+ 974us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:25.961189 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:25.962976 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:25.968606 (+ 5630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:25.969977 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:25.970111 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:25.970344 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:25.971733 (+ 1389us) server_negotiation.cc:299] Negotiation successful
0925 05:05:25.971919 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12608,"thread_start_us":12775,"threads_started":1}
I20260925 05:05:25.974759 16786 tablet_service.cc:1467] Processing CreateTablet for tablet 7c33c361249e498ca21fa4f9318ffd7d (DEFAULT_TABLE table=test-table [id=570889eb08e3468e8d5ce90c2cba3dd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:25.975066 16519 tablet_service.cc:1467] Processing CreateTablet for tablet 7c33c361249e498ca21fa4f9318ffd7d (DEFAULT_TABLE table=test-table [id=570889eb08e3468e8d5ce90c2cba3dd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:25.976097 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:25.960365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54962 (local address 127.30.247.2:37855)
0925 05:05:25.967508 (+ 7143us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:25.967518 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:25.968088 (+ 570us) server_negotiation.cc:407] Connection header received
0925 05:05:25.968317 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:25.968326 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:25.968644 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:25.968830 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:25.969436 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:25.970978 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:25.973357 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:25.974738 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:25.974849 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:25.975030 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:25.975595 (+ 565us) server_negotiation.cc:299] Negotiation successful
0925 05:05:25.975761 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6976,"thread_start_us":6866,"threads_started":1}
I20260925 05:05:25.976329 16519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c33c361249e498ca21fa4f9318ffd7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:25.976337 16786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c33c361249e498ca21fa4f9318ffd7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:25.976814 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:25.964132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:37855 (local address 127.0.0.1:54962)
0925 05:05:25.967945 (+ 3813us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:25.967966 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:25.968098 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:25.968812 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:25.968817 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:25.968842 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:25.969253 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:25.969263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:25.971157 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:25.971162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:25.973184 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:25.973198 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:25.975201 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:25.975207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:25.975380 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:25.975415 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 05:05:25.976339 (+ 924us) client_negotiation.cc:240] Negotiation successful
0925 05:05:25.976469 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3681,"thread_start_us":3543,"threads_started":1}
I20260925 05:05:25.980115 16653 tablet_service.cc:1467] Processing CreateTablet for tablet 7c33c361249e498ca21fa4f9318ffd7d (DEFAULT_TABLE table=test-table [id=570889eb08e3468e8d5ce90c2cba3dd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:25.981585 16653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c33c361249e498ca21fa4f9318ffd7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:25.987449 16869 tablet_bootstrap.cc:492] T 7c33c361249e498ca21fa4f9318ffd7d P f2a5d065a7b349f283a68dab5a30bbff: Bootstrap starting.
I20260925 05:05:25.990721 16869 tablet_bootstrap.cc:654] T 7c33c361249e498ca21fa4f9318ffd7d P f2a5d065a7b349f283a68dab5a30bbff: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:25.990880 16870 tablet_bootstrap.cc:492] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359: Bootstrap starting.
I20260925 05:05:25.992376 16869 log.cc:824] T 7c33c361249e498ca21fa4f9318ffd7d P f2a5d065a7b349f283a68dab5a30bbff: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:25.994905 16870 tablet_bootstrap.cc:654] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:25.996570 16870 log.cc:824] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:25.997258 16872 tablet_bootstrap.cc:492] T 7c33c361249e498ca21fa4f9318ffd7d P 1494216f88f04fc7bf0dd96b46f7f7a5: Bootstrap starting.
I20260925 05:05:25.999919 16872 tablet_bootstrap.cc:654] T 7c33c361249e498ca21fa4f9318ffd7d P 1494216f88f04fc7bf0dd96b46f7f7a5: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:26.001080 16872 log.cc:824] T 7c33c361249e498ca21fa4f9318ffd7d P 1494216f88f04fc7bf0dd96b46f7f7a5: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:26.003265 16870 tablet_bootstrap.cc:492] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359: No bootstrap required, opened a new log
I20260925 05:05:26.003284 16869 tablet_bootstrap.cc:492] T 7c33c361249e498ca21fa4f9318ffd7d P f2a5d065a7b349f283a68dab5a30bbff: No bootstrap required, opened a new log
I20260925 05:05:26.003552 16869 ts_tablet_manager.cc:1397] T 7c33c361249e498ca21fa4f9318ffd7d P f2a5d065a7b349f283a68dab5a30bbff: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260925 05:05:26.003554 16870 ts_tablet_manager.cc:1397] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20260925 05:05:26.003595 16872 tablet_bootstrap.cc:492] T 7c33c361249e498ca21fa4f9318ffd7d P 1494216f88f04fc7bf0dd96b46f7f7a5: No bootstrap required, opened a new log
I20260925 05:05:26.003844 16872 ts_tablet_manager.cc:1397] T 7c33c361249e498ca21fa4f9318ffd7d P 1494216f88f04fc7bf0dd96b46f7f7a5: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 05:05:26.011233 16872 raft_consensus.cc:348] T 7c33c361249e498ca21fa4f9318ffd7d P 1494216f88f04fc7bf0dd96b46f7f7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15398be78d45d2881b81baf4eda359" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45645 } } peers { permanent_uuid: "f2a5d065a7b349f283a68dab5a30bbff" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46855 } } peers { permanent_uuid: "1494216f88f04fc7bf0dd96b46f7f7a5" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 37855 } }
I20260925 05:05:26.011639 16872 raft_consensus.cc:374] T 7c33c361249e498ca21fa4f9318ffd7d P 1494216f88f04fc7bf0dd96b46f7f7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:26.011740 16872 raft_consensus.cc:729] T 7c33c361249e498ca21fa4f9318ffd7d P 1494216f88f04fc7bf0dd96b46f7f7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1494216f88f04fc7bf0dd96b46f7f7a5, State: Initialized, Role: FOLLOWER
I20260925 05:05:26.012387 16872 consensus_queue.cc:260] T 7c33c361249e498ca21fa4f9318ffd7d P 1494216f88f04fc7bf0dd96b46f7f7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15398be78d45d2881b81baf4eda359" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45645 } } peers { permanent_uuid: "f2a5d065a7b349f283a68dab5a30bbff" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46855 } } peers { permanent_uuid: "1494216f88f04fc7bf0dd96b46f7f7a5" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 37855 } }
I20260925 05:05:26.013509 16718 heartbeater.cc:503] Master 127.30.247.62:37371 was elected leader, sending a full tablet report...
I20260925 05:05:26.014252 16872 ts_tablet_manager.cc:1428] T 7c33c361249e498ca21fa4f9318ffd7d P 1494216f88f04fc7bf0dd96b46f7f7a5: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260925 05:05:26.014536 16870 raft_consensus.cc:348] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15398be78d45d2881b81baf4eda359" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45645 } } peers { permanent_uuid: "f2a5d065a7b349f283a68dab5a30bbff" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46855 } } peers { permanent_uuid: "1494216f88f04fc7bf0dd96b46f7f7a5" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 37855 } }
I20260925 05:05:26.014539 16869 raft_consensus.cc:348] T 7c33c361249e498ca21fa4f9318ffd7d P f2a5d065a7b349f283a68dab5a30bbff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15398be78d45d2881b81baf4eda359" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45645 } } peers { permanent_uuid: "f2a5d065a7b349f283a68dab5a30bbff" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46855 } } peers { permanent_uuid: "1494216f88f04fc7bf0dd96b46f7f7a5" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 37855 } }
I20260925 05:05:26.014979 16869 raft_consensus.cc:374] T 7c33c361249e498ca21fa4f9318ffd7d P f2a5d065a7b349f283a68dab5a30bbff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:26.014979 16870 raft_consensus.cc:374] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:26.015087 16870 raft_consensus.cc:729] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a15398be78d45d2881b81baf4eda359, State: Initialized, Role: FOLLOWER
I20260925 05:05:26.015087 16869 raft_consensus.cc:729] T 7c33c361249e498ca21fa4f9318ffd7d P f2a5d065a7b349f283a68dab5a30bbff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2a5d065a7b349f283a68dab5a30bbff, State: Initialized, Role: FOLLOWER
I20260925 05:05:26.015698 16869 consensus_queue.cc:260] T 7c33c361249e498ca21fa4f9318ffd7d P f2a5d065a7b349f283a68dab5a30bbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15398be78d45d2881b81baf4eda359" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45645 } } peers { permanent_uuid: "f2a5d065a7b349f283a68dab5a30bbff" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46855 } } peers { permanent_uuid: "1494216f88f04fc7bf0dd96b46f7f7a5" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 37855 } }
I20260925 05:05:26.015698 16870 consensus_queue.cc:260] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15398be78d45d2881b81baf4eda359" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45645 } } peers { permanent_uuid: "f2a5d065a7b349f283a68dab5a30bbff" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46855 } } peers { permanent_uuid: "1494216f88f04fc7bf0dd96b46f7f7a5" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 37855 } }
I20260925 05:05:26.016918 16851 heartbeater.cc:503] Master 127.30.247.62:37371 was elected leader, sending a full tablet report...
I20260925 05:05:26.018083 16869 ts_tablet_manager.cc:1428] T 7c33c361249e498ca21fa4f9318ffd7d P f2a5d065a7b349f283a68dab5a30bbff: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260925 05:05:26.019254 16870 ts_tablet_manager.cc:1428] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260925 05:05:26.080224 16876 raft_consensus.cc:482] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:05:26.080530 16876 raft_consensus.cc:504] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15398be78d45d2881b81baf4eda359" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45645 } } peers { permanent_uuid: "f2a5d065a7b349f283a68dab5a30bbff" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46855 } } peers { permanent_uuid: "1494216f88f04fc7bf0dd96b46f7f7a5" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 37855 } }
I20260925 05:05:26.082240 16876 leader_election.cc:288] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2a5d065a7b349f283a68dab5a30bbff (127.30.247.1:46855), 1494216f88f04fc7bf0dd96b46f7f7a5 (127.30.247.2:37855)
I20260925 05:05:26.089676 16866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:26.082670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:45861 (local address 127.30.247.1:46855)
0925 05:05:26.082887 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:26.082895 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:26.083104 (+ 209us) server_negotiation.cc:407] Connection header received
0925 05:05:26.083340 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:26.083347 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:26.083431 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:26.083528 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:26.083990 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:26.084968 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:26.087492 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:26.088703 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:26.088875 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:26.088962 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:26.089327 (+ 365us) server_negotiation.cc:299] Negotiation successful
0925 05:05:26.089457 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 05:05:26.089740 16853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:26.082584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:46855 (local address 127.30.247.3:45861)
0925 05:05:26.082972 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:26.083000 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:26.083177 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:26.083597 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:26.083603 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:26.083635 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:26.083836 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:26.083851 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:26.085106 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:26.085112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:26.087332 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:26.087351 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:26.089034 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:26.089038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:26.089165 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:26.089188 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 05:05:26.089260 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 05:05:26.089419 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":2}
I20260925 05:05:26.090117 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:26.082741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:37367 (local address 127.30.247.2:37855)
0925 05:05:26.082942 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:26.082949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:26.083557 (+ 608us) server_negotiation.cc:407] Connection header received
0925 05:05:26.083698 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:26.083704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:26.083784 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:26.083897 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:26.084495 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:26.085359 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:26.087987 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:26.089198 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:26.089361 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:26.089426 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:26.089736 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 05:05:26.089896 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 05:05:26.090281 16878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:26.082648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:37855 (local address 127.30.247.3:37367)
0925 05:05:26.083413 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:26.083430 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:26.083554 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:26.083930 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:26.083935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:26.083960 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:26.084334 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:26.084347 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:26.085516 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:26.085526 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:26.087782 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:26.087800 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:26.089404 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:26.089410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:26.089566 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:26.089603 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 05:05:26.089690 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 05:05:26.089948 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":44,"thread_start_us":413,"threads_started":1}
I20260925 05:05:26.090802 16539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c33c361249e498ca21fa4f9318ffd7d" candidate_uuid: "1a15398be78d45d2881b81baf4eda359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2a5d065a7b349f283a68dab5a30bbff" is_pre_election: true
I20260925 05:05:26.091251 16539 raft_consensus.cc:2454] T 7c33c361249e498ca21fa4f9318ffd7d P f2a5d065a7b349f283a68dab5a30bbff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a15398be78d45d2881b81baf4eda359 in term 0.
I20260925 05:05:26.091851 16739 leader_election.cc:302] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a15398be78d45d2881b81baf4eda359, f2a5d065a7b349f283a68dab5a30bbff; no voters:
I20260925 05:05:26.092231 16876 raft_consensus.cc:2781] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:05:26.092379 16876 raft_consensus.cc:482] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:05:26.092485 16876 raft_consensus.cc:3037] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:26.092422 16673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c33c361249e498ca21fa4f9318ffd7d" candidate_uuid: "1a15398be78d45d2881b81baf4eda359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1494216f88f04fc7bf0dd96b46f7f7a5" is_pre_election: true
I20260925 05:05:26.092926 16673 raft_consensus.cc:2454] T 7c33c361249e498ca21fa4f9318ffd7d P 1494216f88f04fc7bf0dd96b46f7f7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a15398be78d45d2881b81baf4eda359 in term 0.
I20260925 05:05:26.094853 16876 raft_consensus.cc:504] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15398be78d45d2881b81baf4eda359" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45645 } } peers { permanent_uuid: "f2a5d065a7b349f283a68dab5a30bbff" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46855 } } peers { permanent_uuid: "1494216f88f04fc7bf0dd96b46f7f7a5" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 37855 } }
I20260925 05:05:26.095489 16876 leader_election.cc:288] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359 [CANDIDATE]: Term 1 election: Requested vote from peers f2a5d065a7b349f283a68dab5a30bbff (127.30.247.1:46855), 1494216f88f04fc7bf0dd96b46f7f7a5 (127.30.247.2:37855)
I20260925 05:05:26.095955 16673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c33c361249e498ca21fa4f9318ffd7d" candidate_uuid: "1a15398be78d45d2881b81baf4eda359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1494216f88f04fc7bf0dd96b46f7f7a5"
I20260925 05:05:26.095966 16539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c33c361249e498ca21fa4f9318ffd7d" candidate_uuid: "1a15398be78d45d2881b81baf4eda359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2a5d065a7b349f283a68dab5a30bbff"
I20260925 05:05:26.096153 16673 raft_consensus.cc:3037] T 7c33c361249e498ca21fa4f9318ffd7d P 1494216f88f04fc7bf0dd96b46f7f7a5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:26.096153 16539 raft_consensus.cc:3037] T 7c33c361249e498ca21fa4f9318ffd7d P f2a5d065a7b349f283a68dab5a30bbff [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:26.099323 16673 raft_consensus.cc:2454] T 7c33c361249e498ca21fa4f9318ffd7d P 1494216f88f04fc7bf0dd96b46f7f7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a15398be78d45d2881b81baf4eda359 in term 1.
I20260925 05:05:26.099323 16539 raft_consensus.cc:2454] T 7c33c361249e498ca21fa4f9318ffd7d P f2a5d065a7b349f283a68dab5a30bbff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a15398be78d45d2881b81baf4eda359 in term 1.
I20260925 05:05:26.099700 16739 leader_election.cc:302] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a15398be78d45d2881b81baf4eda359, f2a5d065a7b349f283a68dab5a30bbff; no voters:
I20260925 05:05:26.099982 16876 raft_consensus.cc:2781] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:26.100404 16876 raft_consensus.cc:686] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359 [term 1 LEADER]: Becoming Leader. State: Replica: 1a15398be78d45d2881b81baf4eda359, State: Running, Role: LEADER
I20260925 05:05:26.100864 16876 consensus_queue.cc:237] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15398be78d45d2881b81baf4eda359" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45645 } } peers { permanent_uuid: "f2a5d065a7b349f283a68dab5a30bbff" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46855 } } peers { permanent_uuid: "1494216f88f04fc7bf0dd96b46f7f7a5" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 37855 } }
I20260925 05:05:26.105363 16393 catalog_manager.cc:5495] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a15398be78d45d2881b81baf4eda359 (127.30.247.3). New cstate: current_term: 1 leader_uuid: "1a15398be78d45d2881b81baf4eda359" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a15398be78d45d2881b81baf4eda359" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 45645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2a5d065a7b349f283a68dab5a30bbff" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1494216f88f04fc7bf0dd96b46f7f7a5" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 37855 } health_report { overall_health: UNKNOWN } } }
W20260925 05:05:26.131080 16852 tablet.cc:2369] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:05:26.139556 16719 tablet.cc:2369] T 7c33c361249e498ca21fa4f9318ffd7d P 1494216f88f04fc7bf0dd96b46f7f7a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:05:26.145058 16585 tablet.cc:2369] T 7c33c361249e498ca21fa4f9318ffd7d P f2a5d065a7b349f283a68dab5a30bbff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:05:26.157513 16865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:26.152261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41682 (local address 127.30.247.3:45645)
0925 05:05:26.152468 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:26.152472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:26.152490 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:05:26.152679 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:26.152682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:26.152765 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:26.152882 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:05:26.153231 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:26.153936 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:26.155474 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:26.155878 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:26.155974 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:26.156424 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:05:26.156980 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:05:26.157094 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:26.157174 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 05:05:26.157234 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 05:05:26.165201 16539 raft_consensus.cc:1261] T 7c33c361249e498ca21fa4f9318ffd7d P f2a5d065a7b349f283a68dab5a30bbff [term 1 FOLLOWER]: Refusing update from remote peer 1a15398be78d45d2881b81baf4eda359: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:26.165253 16673 raft_consensus.cc:1261] T 7c33c361249e498ca21fa4f9318ffd7d P 1494216f88f04fc7bf0dd96b46f7f7a5 [term 1 FOLLOWER]: Refusing update from remote peer 1a15398be78d45d2881b81baf4eda359: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:26.166064 16879 consensus_queue.cc:1035] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a5d065a7b349f283a68dab5a30bbff" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:26.166422 16876 consensus_queue.cc:1035] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1494216f88f04fc7bf0dd96b46f7f7a5" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 37855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:26.177088 16882 mvcc.cc:205] Tried to move back new op lower bound from 7333120926362906624 to 7333120926112645120. Current Snapshot: MvccSnapshot[applied={T|T < 7333120926362906624}]
I20260925 05:05:26.178287 16884 mvcc.cc:205] Tried to move back new op lower bound from 7333120926362906624 to 7333120926112645120. Current Snapshot: MvccSnapshot[applied={T|T < 7333120926362906624}]
I20260925 05:05:26.188982 16393 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:26.189232 16393 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:26.192729 16393 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d7da84a387447f2824b44ca68e57ab4: Sending DeleteTablet for 3 replicas of tablet 7c33c361249e498ca21fa4f9318ffd7d
I20260925 05:05:26.193814 16653 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c33c361249e498ca21fa4f9318ffd7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:26 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:54962
I20260925 05:05:26.193822 16519 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c33c361249e498ca21fa4f9318ffd7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:26 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:53046
I20260925 05:05:26.193992 16786 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c33c361249e498ca21fa4f9318ffd7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:26 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:41670
I20260925 05:05:26.195848 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 16456
I20260925 05:05:26.205598 16891 tablet_replica.cc:330] T 7c33c361249e498ca21fa4f9318ffd7d P f2a5d065a7b349f283a68dab5a30bbff: stopping tablet replica
I20260925 05:05:26.206245 16891 raft_consensus.cc:2229] T 7c33c361249e498ca21fa4f9318ffd7d P f2a5d065a7b349f283a68dab5a30bbff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:26.206928 16891 raft_consensus.cc:2258] T 7c33c361249e498ca21fa4f9318ffd7d P f2a5d065a7b349f283a68dab5a30bbff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:26.207535 16892 tablet_replica.cc:330] T 7c33c361249e498ca21fa4f9318ffd7d P 1494216f88f04fc7bf0dd96b46f7f7a5: stopping tablet replica
I20260925 05:05:26.209851 16891 ts_tablet_manager.cc:1905] T 7c33c361249e498ca21fa4f9318ffd7d P f2a5d065a7b349f283a68dab5a30bbff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:26.208451 16892 raft_consensus.cc:2229] T 7c33c361249e498ca21fa4f9318ffd7d P 1494216f88f04fc7bf0dd96b46f7f7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:26.210389 16892 raft_consensus.cc:2258] T 7c33c361249e498ca21fa4f9318ffd7d P 1494216f88f04fc7bf0dd96b46f7f7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:26.212322 16892 ts_tablet_manager.cc:1905] T 7c33c361249e498ca21fa4f9318ffd7d P 1494216f88f04fc7bf0dd96b46f7f7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:26.214250 16893 tablet_replica.cc:330] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359: stopping tablet replica
Sep 25 05:05:26 dist-test-slave-5n86 krb5kdc[16350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.1@KRBTEST.COM
I20260925 05:05:26.217955 16893 raft_consensus.cc:2229] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:05:26.218611 16893 raft_consensus.cc:2258] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:26.219028 16892 ts_tablet_manager.cc:1918] T 7c33c361249e498ca21fa4f9318ffd7d P 1494216f88f04fc7bf0dd96b46f7f7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:05:26.219240 16892 log.cc:1196] T 7c33c361249e498ca21fa4f9318ffd7d P 1494216f88f04fc7bf0dd96b46f7f7a5: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-1/wal/wals/7c33c361249e498ca21fa4f9318ffd7d
I20260925 05:05:26.219259 16891 ts_tablet_manager.cc:1918] T 7c33c361249e498ca21fa4f9318ffd7d P f2a5d065a7b349f283a68dab5a30bbff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:05:26.219460 16891 log.cc:1196] T 7c33c361249e498ca21fa4f9318ffd7d P f2a5d065a7b349f283a68dab5a30bbff: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-0/wal/wals/7c33c361249e498ca21fa4f9318ffd7d
I20260925 05:05:26.219868 16892 ts_tablet_manager.cc:1939] T 7c33c361249e498ca21fa4f9318ffd7d P 1494216f88f04fc7bf0dd96b46f7f7a5: Deleting consensus metadata
I20260925 05:05:26.220049 16891 ts_tablet_manager.cc:1939] T 7c33c361249e498ca21fa4f9318ffd7d P f2a5d065a7b349f283a68dab5a30bbff: Deleting consensus metadata
I20260925 05:05:26.221526 16893 ts_tablet_manager.cc:1905] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:26.221680 16382 catalog_manager.cc:4841] TS 1494216f88f04fc7bf0dd96b46f7f7a5 (127.30.247.2:37855): tablet 7c33c361249e498ca21fa4f9318ffd7d (table test-table [id=570889eb08e3468e8d5ce90c2cba3dd1]) successfully deleted
I20260925 05:05:26.221730 16379 catalog_manager.cc:4841] TS f2a5d065a7b349f283a68dab5a30bbff (127.30.247.1:46855): tablet 7c33c361249e498ca21fa4f9318ffd7d (table test-table [id=570889eb08e3468e8d5ce90c2cba3dd1]) successfully deleted
I20260925 05:05:26.228493 16893 ts_tablet_manager.cc:1918] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:05:26.228718 16893 log.cc:1196] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312430481417-31708-0/minicluster-data/ts-2/wal/wals/7c33c361249e498ca21fa4f9318ffd7d
I20260925 05:05:26.229257 16893 ts_tablet_manager.cc:1939] T 7c33c361249e498ca21fa4f9318ffd7d P 1a15398be78d45d2881b81baf4eda359: Deleting consensus metadata
I20260925 05:05:26.229324 16866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:26.197614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53054 (local address 127.30.247.1:46855)
0925 05:05:26.202670 (+ 5056us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:26.202676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:26.202693 (+ 17us) server_negotiation.cc:407] Connection header received
0925 05:05:26.202861 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:26.202870 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:26.202968 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:26.203078 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:26.209025 (+ 5947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:26.210105 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:26.212170 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:26.212804 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:26.212920 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:26.222089 (+ 9169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:26.222105 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:26.222133 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:26.222172 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:26.225925 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:26.226745 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:26.226754 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:26.226757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:26.226914 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:26.227494 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:26.227499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:26.227502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:26.227951 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:26.228322 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:26.228675 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 05:05:26.228905 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4931}
I20260925 05:05:26.230233 16579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:26.230832 16382 catalog_manager.cc:4841] TS 1a15398be78d45d2881b81baf4eda359 (127.30.247.3:45645): tablet 7c33c361249e498ca21fa4f9318ffd7d (table test-table [id=570889eb08e3468e8d5ce90c2cba3dd1]) successfully deleted
I20260925 05:05:26.362138 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 16456
I20260925 05:05:26.386054 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 16590
Sep 25 05:05:26 dist-test-slave-5n86 krb5kdc[16350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.2@KRBTEST.COM
I20260925 05:05:26.406139 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:26.386878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54968 (local address 127.30.247.2:37855)
0925 05:05:26.387070 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:26.387077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:26.387192 (+ 115us) server_negotiation.cc:407] Connection header received
0925 05:05:26.387403 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:26.387410 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:26.387502 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:26.387613 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:26.389229 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:26.390474 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:26.392355 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:26.393187 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:26.393316 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:26.398900 (+ 5584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:26.398912 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:26.398929 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:26.398968 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:26.403025 (+ 4057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:26.403829 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:26.403838 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:26.403842 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:26.404004 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:26.404381 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:26.404387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:26.404390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:26.404911 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:26.405193 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:26.405507 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 05:05:26.405726 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 05:05:26.406804 16713 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:26.498240 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 16590
I20260925 05:05:26.515832 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 16723
Sep 25 05:05:26 dist-test-slave-5n86 krb5kdc[16350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.3@KRBTEST.COM
I20260925 05:05:26.534493 16865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:26.516709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41686 (local address 127.30.247.3:45645)
0925 05:05:26.516900 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:26.516904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:26.516953 (+ 49us) server_negotiation.cc:407] Connection header received
0925 05:05:26.517160 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:26.517164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:26.517240 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:26.517322 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:26.519092 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:26.519939 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:26.521939 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:26.522400 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:26.522504 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:26.528291 (+ 5787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:26.528306 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:26.528325 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:26.528394 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:26.531636 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:26.532288 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:26.532305 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:26.532307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:26.532455 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:26.532929 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:26.532933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:26.532934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:26.533295 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:26.533638 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:26.533869 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 05:05:26.534112 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 05:05:26.535377 16846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:26.656064 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 16723
I20260925 05:05:26.675277 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 16363
I20260925 05:05:26.676519 16425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:26.786794 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 16363
2026-09-25T05:05:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4309 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T05:05:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:26Z Disabled control of system clock
I20260925 05:05:26.859983 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:36367
--webserver_interface=127.30.247.62
--webserver_port=0
--builtin_ntp_servers=127.30.247.20:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:36367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 05:05:27.179700 16902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:27.180048 16902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:27.180202 16902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:27.187282 16902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:27.187377 16902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:27.187448 16902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:27.187539 16902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:27.187602 16902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:27.196674 16902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:36367
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:36367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:27.197901 16902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:27.199612 16902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:27.211092 16907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:27.211122 16910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:27.211603 16908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:27.211651 16902 server_base.cc:1034] running on GCE node
I20260925 05:05:27.212391 16902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:27.213716 16902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:27.214920 16902 hybrid_clock.cc:648] HybridClock initialized: now 1790312727214876 us; error 35 us; skew 500 ppm
I20260925 05:05:27.215277 16902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:05:27.222280 16902 webserver.cc:460] Webserver started at http://127.30.247.62:41291/ using document root <none> and password file <none>
I20260925 05:05:27.222939 16902 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:27.223049 16902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:27.223366 16902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:27.226142 16902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "fec928afb1e74d61b1ed3857f53999ba"
format_stamp: "Formatted at 2026-09-25 05:05:27 on dist-test-slave-5n86"
server_key: "e433da7952b39d56b6e5f3545f6369b8"
server_key_iv: "f8bb69f7bc3d77faa7595f63d770ed60"
server_key_version: "encryptionkey@0"
I20260925 05:05:27.226900 16902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "fec928afb1e74d61b1ed3857f53999ba"
format_stamp: "Formatted at 2026-09-25 05:05:27 on dist-test-slave-5n86"
server_key: "e433da7952b39d56b6e5f3545f6369b8"
server_key_iv: "f8bb69f7bc3d77faa7595f63d770ed60"
server_key_version: "encryptionkey@0"
I20260925 05:05:27.231798 16902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 05:05:27.235143 16916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:27.236522 16902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:05:27.236778 16902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "fec928afb1e74d61b1ed3857f53999ba"
format_stamp: "Formatted at 2026-09-25 05:05:27 on dist-test-slave-5n86"
server_key: "e433da7952b39d56b6e5f3545f6369b8"
server_key_iv: "f8bb69f7bc3d77faa7595f63d770ed60"
server_key_version: "encryptionkey@0"
I20260925 05:05:27.236981 16902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:27.252842 16902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:27.253571 16902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:27.253852 16902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:27.271706 16902 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:36367
I20260925 05:05:27.271725 16967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:36367 every 8 connection(s)
I20260925 05:05:27.273037 16902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:27.276618 16968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:27.281575 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 16902
I20260925 05:05:27.281749 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:05:27.282258 31708 external_mini_cluster.cc:1428] Setting key ce19f0537899b77c9ccfd97e75494392
I20260925 05:05:27.287156 16968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fec928afb1e74d61b1ed3857f53999ba: Bootstrap starting.
I20260925 05:05:27.290570 16968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fec928afb1e74d61b1ed3857f53999ba: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:27.291711 16968 log.cc:824] T 00000000000000000000000000000000 P fec928afb1e74d61b1ed3857f53999ba: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:27.294927 16968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fec928afb1e74d61b1ed3857f53999ba: No bootstrap required, opened a new log
I20260925 05:05:27.296030 16971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:27.284071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45012 (local address 127.30.247.62:36367)
0925 05:05:27.285115 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:27.285125 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:27.285169 (+ 44us) server_negotiation.cc:407] Connection header received
0925 05:05:27.286782 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:27.286803 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:27.287237 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:27.287564 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:27.288106 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:27.289419 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:27.291293 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:27.292060 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:27.292301 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:27.293359 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:27.293371 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:27.293391 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:27.294044 (+ 653us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:27.294180 (+ 136us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:27.294206 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:27.294346 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:27.294534 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 05:05:27.294818 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":399,"threads_started":1}
I20260925 05:05:27.302812 16968 raft_consensus.cc:348] T 00000000000000000000000000000000 P fec928afb1e74d61b1ed3857f53999ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec928afb1e74d61b1ed3857f53999ba" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36367 } }
I20260925 05:05:27.303112 16968 raft_consensus.cc:374] T 00000000000000000000000000000000 P fec928afb1e74d61b1ed3857f53999ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:27.303184 16968 raft_consensus.cc:729] T 00000000000000000000000000000000 P fec928afb1e74d61b1ed3857f53999ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec928afb1e74d61b1ed3857f53999ba, State: Initialized, Role: FOLLOWER
I20260925 05:05:27.303807 16968 consensus_queue.cc:260] T 00000000000000000000000000000000 P fec928afb1e74d61b1ed3857f53999ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec928afb1e74d61b1ed3857f53999ba" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36367 } }
I20260925 05:05:27.304101 16968 raft_consensus.cc:388] T 00000000000000000000000000000000 P fec928afb1e74d61b1ed3857f53999ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:27.304244 16968 raft_consensus.cc:482] T 00000000000000000000000000000000 P fec928afb1e74d61b1ed3857f53999ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:27.304409 16968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fec928afb1e74d61b1ed3857f53999ba [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:27.306499 16968 raft_consensus.cc:504] T 00000000000000000000000000000000 P fec928afb1e74d61b1ed3857f53999ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec928afb1e74d61b1ed3857f53999ba" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36367 } }
I20260925 05:05:27.307077 16968 leader_election.cc:302] T 00000000000000000000000000000000 P fec928afb1e74d61b1ed3857f53999ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fec928afb1e74d61b1ed3857f53999ba; no voters:
I20260925 05:05:27.307654 16968 leader_election.cc:288] T 00000000000000000000000000000000 P fec928afb1e74d61b1ed3857f53999ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:27.307797 16973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fec928afb1e74d61b1ed3857f53999ba [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:27.308470 16973 raft_consensus.cc:686] T 00000000000000000000000000000000 P fec928afb1e74d61b1ed3857f53999ba [term 1 LEADER]: Becoming Leader. State: Replica: fec928afb1e74d61b1ed3857f53999ba, State: Running, Role: LEADER
I20260925 05:05:27.308985 16973 consensus_queue.cc:237] T 00000000000000000000000000000000 P fec928afb1e74d61b1ed3857f53999ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec928afb1e74d61b1ed3857f53999ba" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36367 } }
I20260925 05:05:27.309485 16968 sys_catalog.cc:564] T 00000000000000000000000000000000 P fec928afb1e74d61b1ed3857f53999ba [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:27.311847 16974 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec928afb1e74d61b1ed3857f53999ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fec928afb1e74d61b1ed3857f53999ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec928afb1e74d61b1ed3857f53999ba" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36367 } } }
I20260925 05:05:27.312108 16975 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec928afb1e74d61b1ed3857f53999ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader fec928afb1e74d61b1ed3857f53999ba. Latest consensus state: current_term: 1 leader_uuid: "fec928afb1e74d61b1ed3857f53999ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec928afb1e74d61b1ed3857f53999ba" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 36367 } } }
I20260925 05:05:27.312373 16975 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec928afb1e74d61b1ed3857f53999ba [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:27.312273 16974 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec928afb1e74d61b1ed3857f53999ba [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:27.315762 16982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:27.321336 16982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:27.322099 16990 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 05:05:27.330567 16982 catalog_manager.cc:1347] Generated new cluster ID: a8a17e20a7ec4716b5ccab51eb66890e
I20260925 05:05:27.330677 16982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:05:27.355510 16982 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:27.356774 16982 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:27.364004 16982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fec928afb1e74d61b1ed3857f53999ba: Generated new TSK 0
I20260925 05:05:27.364791 16982 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:05:27.376273 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:0
--local_ip_for_outbound_sockets=127.30.247.1
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:36367
--builtin_ntp_servers=127.30.247.20:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 05:05:27.729097 16992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:27.729343 16992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:27.729480 16992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:27.736459 16992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:27.736613 16992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:27.736773 16992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:05:27.746253 16992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:36367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:27.747447 16992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:27.749209 16992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:27.761686 16998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:27.761754 17000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:27.761776 16997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:27.762926 16992 server_base.cc:1034] running on GCE node
I20260925 05:05:27.763499 16992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:27.764436 16992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:27.765664 16992 hybrid_clock.cc:648] HybridClock initialized: now 1790312727765638 us; error 64 us; skew 500 ppm
I20260925 05:05:27.765973 16992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:05:27.768200 16992 webserver.cc:460] Webserver started at http://127.30.247.1:46645/ using document root <none> and password file <none>
I20260925 05:05:27.768956 16992 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:27.769083 16992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:27.769388 16992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:27.772171 16992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-0/data/instance:
uuid: "1b1cd998c1cc4e06af8af72cd1c8f157"
format_stamp: "Formatted at 2026-09-25 05:05:27 on dist-test-slave-5n86"
server_key: "aae0a6fe5b68a34e30308cd97a6afcd9"
server_key_iv: "735d4cdd6cf86df40b45109f3fb70c38"
server_key_version: "encryptionkey@0"
I20260925 05:05:27.772959 16992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance:
uuid: "1b1cd998c1cc4e06af8af72cd1c8f157"
format_stamp: "Formatted at 2026-09-25 05:05:27 on dist-test-slave-5n86"
server_key: "aae0a6fe5b68a34e30308cd97a6afcd9"
server_key_iv: "735d4cdd6cf86df40b45109f3fb70c38"
server_key_version: "encryptionkey@0"
I20260925 05:05:27.777752 16992 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 05:05:27.780949 17006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:27.782187 16992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 05:05:27.782351 16992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "1b1cd998c1cc4e06af8af72cd1c8f157"
format_stamp: "Formatted at 2026-09-25 05:05:27 on dist-test-slave-5n86"
server_key: "aae0a6fe5b68a34e30308cd97a6afcd9"
server_key_iv: "735d4cdd6cf86df40b45109f3fb70c38"
server_key_version: "encryptionkey@0"
I20260925 05:05:27.782562 16992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:27.815833 16992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:27.816720 16992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:27.817010 16992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:27.817926 16992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:27.819273 16992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:27.819386 16992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:27.819505 16992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:27.819597 16992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:27.851436 16992 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:45619
I20260925 05:05:27.851531 17118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:45619 every 8 connection(s)
I20260925 05:05:27.852787 16992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:05:27.854252 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 16992
I20260925 05:05:27.854449 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance
I20260925 05:05:27.854977 31708 external_mini_cluster.cc:1428] Setting key 80ca8cd4714289641a1aa6f35040d6f3
I20260925 05:05:27.858163 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:0
--local_ip_for_outbound_sockets=127.30.247.2
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:36367
--builtin_ntp_servers=127.30.247.20:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 05:05:27.868255 17121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:27.855569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:38835 (local address 127.30.247.62:36367)
0925 05:05:27.856084 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:27.856088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:27.857608 (+ 1520us) server_negotiation.cc:407] Connection header received
0925 05:05:27.860431 (+ 2823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:27.860435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:27.860498 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:27.860643 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:27.862012 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:27.862700 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:27.865228 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:27.865644 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:27.865753 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:27.867285 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:27.867294 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:27.867299 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:27.867324 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:27.867388 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:27.867407 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:27.867511 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:27.867711 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 05:05:27.867826 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":245,"threads_started":1}
I20260925 05:05:27.869019 17122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:27.856260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36367 (local address 127.30.247.1:38835)
0925 05:05:27.857434 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:27.857483 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:27.860108 (+ 2625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:27.860902 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:27.860916 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:27.861035 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:27.861820 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:27.861846 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:27.862876 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:27.862884 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:27.865077 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:27.865099 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:27.865875 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:27.865884 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:27.866215 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:27.866968 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:05:27.866970 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:27.867008 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:05:27.867081 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:27.867505 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:27.867512 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:27.867518 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 05:05:27.867653 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 05:05:27.867957 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":395,"threads_started":1}
I20260925 05:05:27.870488 17119 heartbeater.cc:347] Connected to a master server at 127.30.247.62:36367
I20260925 05:05:27.870790 17119 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:27.871500 17119 heartbeater.cc:511] Master 127.30.247.62:36367 requested a full tablet report, sending...
I20260925 05:05:27.873477 16933 ts_manager.cc:194] Registered new tserver with Master: 1b1cd998c1cc4e06af8af72cd1c8f157 (127.30.247.1:45619)
I20260925 05:05:27.874868 16933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.247.1:38835
W20260925 05:05:28.182120 17123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:28.182372 17123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:28.182529 17123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:28.189432 17123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:28.189539 17123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:28.189705 17123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:05:28.198328 17123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:36367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:28.199467 17123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:28.201184 17123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:28.213860 17129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:28.213904 17131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:28.213950 17128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:28.224038 17123 server_base.cc:1034] running on GCE node
I20260925 05:05:28.224761 17123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:28.225666 17123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:28.226866 17123 hybrid_clock.cc:648] HybridClock initialized: now 1790312728226816 us; error 40 us; skew 500 ppm
I20260925 05:05:28.227180 17123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:05:28.229363 17123 webserver.cc:460] Webserver started at http://127.30.247.2:39333/ using document root <none> and password file <none>
I20260925 05:05:28.230018 17123 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:28.230137 17123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:28.230437 17123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:28.233165 17123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-1/data/instance:
uuid: "5aa3fd5c69244b17b0030e272e139dba"
format_stamp: "Formatted at 2026-09-25 05:05:28 on dist-test-slave-5n86"
server_key: "c4acd9b84942ebf2c8c793b702751ae6"
server_key_iv: "4f2b48a19033ccaf935bb3ccbc65c191"
server_key_version: "encryptionkey@0"
I20260925 05:05:28.233922 17123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance:
uuid: "5aa3fd5c69244b17b0030e272e139dba"
format_stamp: "Formatted at 2026-09-25 05:05:28 on dist-test-slave-5n86"
server_key: "c4acd9b84942ebf2c8c793b702751ae6"
server_key_iv: "4f2b48a19033ccaf935bb3ccbc65c191"
server_key_version: "encryptionkey@0"
I20260925 05:05:28.238730 17123 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 05:05:28.241920 17137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:28.243110 17123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 05:05:28.243279 17123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "5aa3fd5c69244b17b0030e272e139dba"
format_stamp: "Formatted at 2026-09-25 05:05:28 on dist-test-slave-5n86"
server_key: "c4acd9b84942ebf2c8c793b702751ae6"
server_key_iv: "4f2b48a19033ccaf935bb3ccbc65c191"
server_key_version: "encryptionkey@0"
I20260925 05:05:28.243485 17123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:28.254259 17123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:28.254949 17123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:28.255224 17123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:28.256095 17123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:28.257543 17123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:28.257653 17123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:28.257754 17123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:28.257834 17123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:28.290241 17123 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:36563
I20260925 05:05:28.290323 17249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:36563 every 8 connection(s)
I20260925 05:05:28.291457 17123 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:05:28.293735 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 17123
I20260925 05:05:28.293910 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance
I20260925 05:05:28.294451 31708 external_mini_cluster.cc:1428] Setting key ee86f3926368c1d8e2edb99d285f30cc
I20260925 05:05:28.297325 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:0
--local_ip_for_outbound_sockets=127.30.247.3
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:36367
--builtin_ntp_servers=127.30.247.20:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 05:05:28.303898 17121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:28.293909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:54587 (local address 127.30.247.62:36367)
0925 05:05:28.294065 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:28.294070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:28.295778 (+ 1708us) server_negotiation.cc:407] Connection header received
0925 05:05:28.297805 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:28.297811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:28.297892 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:28.298005 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:28.298998 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.299695 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.301279 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.301696 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.301840 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:28.303126 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:28.303131 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:28.303140 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:28.303157 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:28.303187 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:28.303198 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:28.303288 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:28.303494 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 05:05:28.303587 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 05:05:28.304867 17252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:28.294584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36367 (local address 127.30.247.2:54587)
0925 05:05:28.295613 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:28.295665 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:28.297563 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:28.298214 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:28.298222 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:28.298318 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:28.298845 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:28.298862 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.299862 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.299871 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:28.301142 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:28.301154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.301919 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.301926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:28.302160 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:28.302836 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:05:28.302839 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:28.302878 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:05:28.302948 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:28.303293 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:28.303306 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:28.303314 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 05:05:28.303490 (+ 176us) client_negotiation.cc:240] Negotiation successful
0925 05:05:28.303843 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":294,"threads_started":1}
I20260925 05:05:28.306147 17250 heartbeater.cc:347] Connected to a master server at 127.30.247.62:36367
I20260925 05:05:28.306435 17250 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:28.307258 17250 heartbeater.cc:511] Master 127.30.247.62:36367 requested a full tablet report, sending...
I20260925 05:05:28.308768 16933 ts_manager.cc:194] Registered new tserver with Master: 5aa3fd5c69244b17b0030e272e139dba (127.30.247.2:36563)
I20260925 05:05:28.309980 16933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.247.2:54587
W20260925 05:05:28.610069 17253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:28.610375 17253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:28.610463 17253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:28.617527 17253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:28.617609 17253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:28.617713 17253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:05:28.626367 17253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:36367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:28.627486 17253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:28.629151 17253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:28.642031 17258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:28.642042 17261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:28.642168 17259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:28.642958 17253 server_base.cc:1034] running on GCE node
I20260925 05:05:28.643555 17253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:28.644435 17253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:28.645658 17253 hybrid_clock.cc:648] HybridClock initialized: now 1790312728645634 us; error 57 us; skew 500 ppm
I20260925 05:05:28.646000 17253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:05:28.648156 17253 webserver.cc:460] Webserver started at http://127.30.247.3:34971/ using document root <none> and password file <none>
I20260925 05:05:28.648878 17253 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:28.649001 17253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:28.649298 17253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:28.651983 17253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-2/data/instance:
uuid: "198755afa3c849e583d33e1c5f4ce99a"
format_stamp: "Formatted at 2026-09-25 05:05:28 on dist-test-slave-5n86"
server_key: "92cc94629e4c5bf0824dd9ef9373ea85"
server_key_iv: "26392981369b43b6b2c4c0d4607f7594"
server_key_version: "encryptionkey@0"
I20260925 05:05:28.652806 17253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance:
uuid: "198755afa3c849e583d33e1c5f4ce99a"
format_stamp: "Formatted at 2026-09-25 05:05:28 on dist-test-slave-5n86"
server_key: "92cc94629e4c5bf0824dd9ef9373ea85"
server_key_iv: "26392981369b43b6b2c4c0d4607f7594"
server_key_version: "encryptionkey@0"
I20260925 05:05:28.658027 17253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 05:05:28.661249 17267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:28.662444 17253 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 05:05:28.662606 17253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "198755afa3c849e583d33e1c5f4ce99a"
format_stamp: "Formatted at 2026-09-25 05:05:28 on dist-test-slave-5n86"
server_key: "92cc94629e4c5bf0824dd9ef9373ea85"
server_key_iv: "26392981369b43b6b2c4c0d4607f7594"
server_key_version: "encryptionkey@0"
I20260925 05:05:28.662832 17253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:28.681279 17253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:28.682018 17253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:28.682313 17253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:28.683205 17253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:28.684646 17253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:28.684765 17253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:28.684885 17253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:28.684974 17253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:28.716814 17253 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:46081
I20260925 05:05:28.716814 17379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:46081 every 8 connection(s)
I20260925 05:05:28.718050 17253 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:05:28.721396 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 17253
I20260925 05:05:28.721561 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance
I20260925 05:05:28.722218 31708 external_mini_cluster.cc:1428] Setting key b8e6be48b46671daa867f3c5b959c0af
I20260925 05:05:28.730890 17121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:28.720968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:34273 (local address 127.30.247.62:36367)
0925 05:05:28.721207 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:28.721214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:28.722786 (+ 1572us) server_negotiation.cc:407] Connection header received
0925 05:05:28.724631 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:28.724636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:28.724731 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:28.724843 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:28.725852 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.726590 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.728098 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.728513 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.728697 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:28.730026 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:28.730031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:28.730035 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:28.730053 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:28.730084 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:28.730097 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:28.730199 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:28.730436 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 05:05:28.730537 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 05:05:28.731866 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:28.721494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:36367 (local address 127.30.247.3:34273)
0925 05:05:28.722608 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:28.722672 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:28.724350 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:28.725061 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:28.725068 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:28.725152 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:28.725697 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:28.725713 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.726751 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.726756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:28.727961 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:28.727973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.728765 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.728772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:28.728994 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:28.729646 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:05:28.729649 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:28.729693 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:05:28.729774 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:28.730202 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:28.730217 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:28.730227 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 05:05:28.730408 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 05:05:28.730785 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":310,"threads_started":1}
I20260925 05:05:28.733156 17380 heartbeater.cc:347] Connected to a master server at 127.30.247.62:36367
I20260925 05:05:28.733418 17380 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:28.734114 17380 heartbeater.cc:511] Master 127.30.247.62:36367 requested a full tablet report, sending...
I20260925 05:05:28.735440 16933 ts_manager.cc:194] Registered new tserver with Master: 198755afa3c849e583d33e1c5f4ce99a (127.30.247.3:46081)
I20260925 05:05:28.736582 16933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.247.3:34273
I20260925 05:05:28.737352 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:05:28.747123 17121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:28.741302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45016 (local address 127.30.247.62:36367)
0925 05:05:28.741495 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:28.741504 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:28.742018 (+ 514us) server_negotiation.cc:407] Connection header received
0925 05:05:28.742190 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:28.742194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:28.742258 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:28.742337 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:28.742882 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.743548 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.745051 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.745456 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.745544 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:28.746501 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:28.746505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:28.746508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:28.746523 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:28.746551 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:28.746562 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:28.746628 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:28.746770 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 05:05:28.746858 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 05:05:28.754721 16932 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:05:28.764020 16932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:05:28.791220 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:28.777332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41602 (local address 127.30.247.3:46081)
0925 05:05:28.781160 (+ 3828us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:28.781168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:28.781202 (+ 34us) server_negotiation.cc:407] Connection header received
0925 05:05:28.781300 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:28.781307 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:28.781566 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:28.782565 (+ 999us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:28.782911 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.784859 (+ 1948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.788429 (+ 3570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.790020 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.790172 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:28.790456 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:28.790679 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 05:05:28.790853 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3621,"thread_start_us":3435,"threads_started":1}
I20260925 05:05:28.791198 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:28.776724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:46081 (local address 127.0.0.1:41602)
0925 05:05:28.777958 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:28.777979 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:28.778115 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:28.781805 (+ 3690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:28.781812 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:28.781845 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:28.782304 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:28.782323 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.785004 (+ 2681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.785009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:28.788041 (+ 3032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:28.788061 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.790198 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.790205 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:28.790438 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:28.790493 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 05:05:28.790641 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 05:05:28.790846 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1060,"thread_start_us":13091,"threads_started":1}
I20260925 05:05:28.795449 17315 tablet_service.cc:1467] Processing CreateTablet for tablet 06504d4370054d8d9b25752556cb8fbf (DEFAULT_TABLE table=test-table [id=fb7114f7513449b79591a93a3b161cfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:28.796886 17315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06504d4370054d8d9b25752556cb8fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:28.797588 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:28.776041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:36563 (local address 127.0.0.1:59176)
0925 05:05:28.776952 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:28.776993 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:28.777180 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:28.790538 (+ 13358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:28.790543 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:28.790567 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:28.791000 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:28.791014 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.793072 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.793079 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:28.794975 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:28.794987 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.796808 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.796814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:28.796994 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:28.797026 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 05:05:28.797125 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 05:05:28.797260 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":60,"thread_start_us":418,"threads_started":1}
I20260925 05:05:28.797719 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:28.776349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59176 (local address 127.30.247.2:36563)
0925 05:05:28.789817 (+ 13468us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:28.789827 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:28.789862 (+ 35us) server_negotiation.cc:407] Connection header received
0925 05:05:28.789974 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:28.789983 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:28.790250 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:28.791196 (+ 946us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:28.791471 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.792963 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.795156 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.796598 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.796739 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:28.797034 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:28.797175 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 05:05:28.797350 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13277,"thread_start_us":13128,"threads_started":1}
I20260925 05:05:28.799454 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:28.777355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:45619 (local address 127.0.0.1:43430)
0925 05:05:28.790052 (+ 12697us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:28.790070 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:28.790282 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:28.791147 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:28.791150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:28.791173 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:28.791566 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:28.791579 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.793135 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.793139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:28.796813 (+ 3674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:28.796827 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.798647 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.798654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:28.798820 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:28.798859 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 05:05:28.798954 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 05:05:28.799076 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12587,"mutex_wait_us":129,"thread_start_us":825,"threads_started":1}
I20260925 05:05:28.799481 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:28.777965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43430 (local address 127.30.247.1:45619)
0925 05:05:28.789138 (+ 11173us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:28.789151 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:28.790165 (+ 1014us) server_negotiation.cc:407] Connection header received
0925 05:05:28.790483 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:28.790491 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:28.790722 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:28.790891 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:28.791751 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.792866 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.797035 (+ 4169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.798445 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.798585 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:28.798821 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:28.799008 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 05:05:28.799138 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10945,"thread_start_us":11172,"threads_started":1}
I20260925 05:05:28.801407 17185 tablet_service.cc:1467] Processing CreateTablet for tablet 06504d4370054d8d9b25752556cb8fbf (DEFAULT_TABLE table=test-table [id=fb7114f7513449b79591a93a3b161cfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:28.802582 17185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06504d4370054d8d9b25752556cb8fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:28.803140 17054 tablet_service.cc:1467] Processing CreateTablet for tablet 06504d4370054d8d9b25752556cb8fbf (DEFAULT_TABLE table=test-table [id=fb7114f7513449b79591a93a3b161cfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:28.804447 17054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06504d4370054d8d9b25752556cb8fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:28.813714 17398 tablet_bootstrap.cc:492] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba: Bootstrap starting.
I20260925 05:05:28.817425 17398 tablet_bootstrap.cc:654] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:28.818393 17398 log.cc:824] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:28.820794 17401 tablet_bootstrap.cc:492] T 06504d4370054d8d9b25752556cb8fbf P 1b1cd998c1cc4e06af8af72cd1c8f157: Bootstrap starting.
I20260925 05:05:28.821277 17398 tablet_bootstrap.cc:492] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba: No bootstrap required, opened a new log
I20260925 05:05:28.821524 17398 ts_tablet_manager.cc:1397] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 05:05:28.823515 17401 tablet_bootstrap.cc:654] T 06504d4370054d8d9b25752556cb8fbf P 1b1cd998c1cc4e06af8af72cd1c8f157: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:28.824410 17399 tablet_bootstrap.cc:492] T 06504d4370054d8d9b25752556cb8fbf P 198755afa3c849e583d33e1c5f4ce99a: Bootstrap starting.
I20260925 05:05:28.824703 17401 log.cc:824] T 06504d4370054d8d9b25752556cb8fbf P 1b1cd998c1cc4e06af8af72cd1c8f157: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:28.827574 17401 tablet_bootstrap.cc:492] T 06504d4370054d8d9b25752556cb8fbf P 1b1cd998c1cc4e06af8af72cd1c8f157: No bootstrap required, opened a new log
I20260925 05:05:28.827756 17399 tablet_bootstrap.cc:654] T 06504d4370054d8d9b25752556cb8fbf P 198755afa3c849e583d33e1c5f4ce99a: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:28.827836 17401 ts_tablet_manager.cc:1397] T 06504d4370054d8d9b25752556cb8fbf P 1b1cd998c1cc4e06af8af72cd1c8f157: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 05:05:28.828924 17399 log.cc:824] T 06504d4370054d8d9b25752556cb8fbf P 198755afa3c849e583d33e1c5f4ce99a: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:28.830343 17398 raft_consensus.cc:348] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa3fd5c69244b17b0030e272e139dba" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36563 } } peers { permanent_uuid: "1b1cd998c1cc4e06af8af72cd1c8f157" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45619 } } peers { permanent_uuid: "198755afa3c849e583d33e1c5f4ce99a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 46081 } }
I20260925 05:05:28.830749 17398 raft_consensus.cc:374] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:28.830850 17398 raft_consensus.cc:729] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa3fd5c69244b17b0030e272e139dba, State: Initialized, Role: FOLLOWER
I20260925 05:05:28.831470 17398 consensus_queue.cc:260] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa3fd5c69244b17b0030e272e139dba" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36563 } } peers { permanent_uuid: "1b1cd998c1cc4e06af8af72cd1c8f157" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45619 } } peers { permanent_uuid: "198755afa3c849e583d33e1c5f4ce99a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 46081 } }
I20260925 05:05:28.831856 17399 tablet_bootstrap.cc:492] T 06504d4370054d8d9b25752556cb8fbf P 198755afa3c849e583d33e1c5f4ce99a: No bootstrap required, opened a new log
I20260925 05:05:28.832180 17399 ts_tablet_manager.cc:1397] T 06504d4370054d8d9b25752556cb8fbf P 198755afa3c849e583d33e1c5f4ce99a: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260925 05:05:28.832625 17250 heartbeater.cc:503] Master 127.30.247.62:36367 was elected leader, sending a full tablet report...
I20260925 05:05:28.833281 17398 ts_tablet_manager.cc:1428] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260925 05:05:28.837384 17401 raft_consensus.cc:348] T 06504d4370054d8d9b25752556cb8fbf P 1b1cd998c1cc4e06af8af72cd1c8f157 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa3fd5c69244b17b0030e272e139dba" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36563 } } peers { permanent_uuid: "1b1cd998c1cc4e06af8af72cd1c8f157" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45619 } } peers { permanent_uuid: "198755afa3c849e583d33e1c5f4ce99a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 46081 } }
I20260925 05:05:28.837790 17401 raft_consensus.cc:374] T 06504d4370054d8d9b25752556cb8fbf P 1b1cd998c1cc4e06af8af72cd1c8f157 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:28.837903 17401 raft_consensus.cc:729] T 06504d4370054d8d9b25752556cb8fbf P 1b1cd998c1cc4e06af8af72cd1c8f157 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b1cd998c1cc4e06af8af72cd1c8f157, State: Initialized, Role: FOLLOWER
I20260925 05:05:28.838497 17401 consensus_queue.cc:260] T 06504d4370054d8d9b25752556cb8fbf P 1b1cd998c1cc4e06af8af72cd1c8f157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa3fd5c69244b17b0030e272e139dba" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36563 } } peers { permanent_uuid: "1b1cd998c1cc4e06af8af72cd1c8f157" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45619 } } peers { permanent_uuid: "198755afa3c849e583d33e1c5f4ce99a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 46081 } }
I20260925 05:05:28.839900 17119 heartbeater.cc:503] Master 127.30.247.62:36367 was elected leader, sending a full tablet report...
I20260925 05:05:28.840476 17401 ts_tablet_manager.cc:1428] T 06504d4370054d8d9b25752556cb8fbf P 1b1cd998c1cc4e06af8af72cd1c8f157: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260925 05:05:28.841903 17399 raft_consensus.cc:348] T 06504d4370054d8d9b25752556cb8fbf P 198755afa3c849e583d33e1c5f4ce99a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa3fd5c69244b17b0030e272e139dba" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36563 } } peers { permanent_uuid: "1b1cd998c1cc4e06af8af72cd1c8f157" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45619 } } peers { permanent_uuid: "198755afa3c849e583d33e1c5f4ce99a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 46081 } }
I20260925 05:05:28.842322 17399 raft_consensus.cc:374] T 06504d4370054d8d9b25752556cb8fbf P 198755afa3c849e583d33e1c5f4ce99a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:28.842419 17399 raft_consensus.cc:729] T 06504d4370054d8d9b25752556cb8fbf P 198755afa3c849e583d33e1c5f4ce99a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 198755afa3c849e583d33e1c5f4ce99a, State: Initialized, Role: FOLLOWER
I20260925 05:05:28.842990 17399 consensus_queue.cc:260] T 06504d4370054d8d9b25752556cb8fbf P 198755afa3c849e583d33e1c5f4ce99a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa3fd5c69244b17b0030e272e139dba" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36563 } } peers { permanent_uuid: "1b1cd998c1cc4e06af8af72cd1c8f157" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45619 } } peers { permanent_uuid: "198755afa3c849e583d33e1c5f4ce99a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 46081 } }
I20260925 05:05:28.844013 17380 heartbeater.cc:503] Master 127.30.247.62:36367 was elected leader, sending a full tablet report...
I20260925 05:05:28.844636 17399 ts_tablet_manager.cc:1428] T 06504d4370054d8d9b25752556cb8fbf P 198755afa3c849e583d33e1c5f4ce99a: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260925 05:05:28.854992 17120 tablet.cc:2369] T 06504d4370054d8d9b25752556cb8fbf P 1b1cd998c1cc4e06af8af72cd1c8f157: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:05:28.879000 17404 raft_consensus.cc:482] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:05:28.879314 17404 raft_consensus.cc:504] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa3fd5c69244b17b0030e272e139dba" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36563 } } peers { permanent_uuid: "1b1cd998c1cc4e06af8af72cd1c8f157" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45619 } } peers { permanent_uuid: "198755afa3c849e583d33e1c5f4ce99a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 46081 } }
I20260925 05:05:28.881125 17404 leader_election.cc:288] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b1cd998c1cc4e06af8af72cd1c8f157 (127.30.247.1:45619), 198755afa3c849e583d33e1c5f4ce99a (127.30.247.3:46081)
I20260925 05:05:28.890014 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:28.881747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:59395 (local address 127.30.247.3:46081)
0925 05:05:28.881966 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:28.881973 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:28.882627 (+ 654us) server_negotiation.cc:407] Connection header received
0925 05:05:28.882778 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:28.882787 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:28.882859 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:28.882944 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:28.883579 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.884648 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.887607 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.888862 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.889018 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:28.889084 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:28.889639 (+ 555us) server_negotiation.cc:299] Negotiation successful
0925 05:05:28.889767 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 05:05:28.890038 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:28.881614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:34565 (local address 127.30.247.1:45619)
0925 05:05:28.882031 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:28.882041 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:28.882319 (+ 278us) server_negotiation.cc:407] Connection header received
0925 05:05:28.882568 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:28.882572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:28.882708 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:28.882803 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:28.883472 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.884632 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.887608 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.888861 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.889015 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:28.889084 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:28.889664 (+ 580us) server_negotiation.cc:299] Negotiation successful
0925 05:05:28.889789 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229}
I20260925 05:05:28.890149 17408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:28.881565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:46081 (local address 127.30.247.2:59395)
0925 05:05:28.882528 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:28.882542 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:28.882650 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:28.882986 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:28.882990 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:28.883010 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:28.883421 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:28.883433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.884781 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.884786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:28.887370 (+ 2584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:28.887386 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.889117 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.889123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:28.889432 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:28.889496 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 05:05:28.889584 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 05:05:28.889737 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":218,"thread_start_us":675,"threads_started":1}
I20260925 05:05:28.890182 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:28.881446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:45619 (local address 127.30.247.2:34565)
0925 05:05:28.882188 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:28.882217 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:28.882403 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:28.882820 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:28.882823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:28.882841 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:28.883293 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:28.883312 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.884780 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.884786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:28.887370 (+ 2584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:28.887385 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.889109 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.889116 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:28.889429 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:28.889489 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 05:05:28.889585 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 05:05:28.889737 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":218,"thread_start_us":511,"threads_started":1}
I20260925 05:05:28.891232 17074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06504d4370054d8d9b25752556cb8fbf" candidate_uuid: "5aa3fd5c69244b17b0030e272e139dba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b1cd998c1cc4e06af8af72cd1c8f157" is_pre_election: true
I20260925 05:05:28.891296 17335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06504d4370054d8d9b25752556cb8fbf" candidate_uuid: "5aa3fd5c69244b17b0030e272e139dba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "198755afa3c849e583d33e1c5f4ce99a" is_pre_election: true
I20260925 05:05:28.891775 17074 raft_consensus.cc:2454] T 06504d4370054d8d9b25752556cb8fbf P 1b1cd998c1cc4e06af8af72cd1c8f157 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aa3fd5c69244b17b0030e272e139dba in term 0.
I20260925 05:05:28.891791 17335 raft_consensus.cc:2454] T 06504d4370054d8d9b25752556cb8fbf P 198755afa3c849e583d33e1c5f4ce99a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aa3fd5c69244b17b0030e272e139dba in term 0.
I20260925 05:05:28.892416 17138 leader_election.cc:302] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 198755afa3c849e583d33e1c5f4ce99a, 5aa3fd5c69244b17b0030e272e139dba; no voters:
I20260925 05:05:28.892803 17404 raft_consensus.cc:2781] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:05:28.892954 17404 raft_consensus.cc:482] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:05:28.893064 17404 raft_consensus.cc:3037] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:28.895147 17404 raft_consensus.cc:504] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa3fd5c69244b17b0030e272e139dba" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36563 } } peers { permanent_uuid: "1b1cd998c1cc4e06af8af72cd1c8f157" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45619 } } peers { permanent_uuid: "198755afa3c849e583d33e1c5f4ce99a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 46081 } }
I20260925 05:05:28.895700 17404 leader_election.cc:288] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba [CANDIDATE]: Term 1 election: Requested vote from peers 1b1cd998c1cc4e06af8af72cd1c8f157 (127.30.247.1:45619), 198755afa3c849e583d33e1c5f4ce99a (127.30.247.3:46081)
I20260925 05:05:28.896086 17074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06504d4370054d8d9b25752556cb8fbf" candidate_uuid: "5aa3fd5c69244b17b0030e272e139dba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b1cd998c1cc4e06af8af72cd1c8f157"
I20260925 05:05:28.896111 17335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06504d4370054d8d9b25752556cb8fbf" candidate_uuid: "5aa3fd5c69244b17b0030e272e139dba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "198755afa3c849e583d33e1c5f4ce99a"
I20260925 05:05:28.896282 17074 raft_consensus.cc:3037] T 06504d4370054d8d9b25752556cb8fbf P 1b1cd998c1cc4e06af8af72cd1c8f157 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:28.896315 17335 raft_consensus.cc:3037] T 06504d4370054d8d9b25752556cb8fbf P 198755afa3c849e583d33e1c5f4ce99a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:28.898547 17074 raft_consensus.cc:2454] T 06504d4370054d8d9b25752556cb8fbf P 1b1cd998c1cc4e06af8af72cd1c8f157 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aa3fd5c69244b17b0030e272e139dba in term 1.
I20260925 05:05:28.898908 17335 raft_consensus.cc:2454] T 06504d4370054d8d9b25752556cb8fbf P 198755afa3c849e583d33e1c5f4ce99a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aa3fd5c69244b17b0030e272e139dba in term 1.
I20260925 05:05:28.898926 17139 leader_election.cc:302] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b1cd998c1cc4e06af8af72cd1c8f157, 5aa3fd5c69244b17b0030e272e139dba; no voters:
I20260925 05:05:28.899216 17404 raft_consensus.cc:2781] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:28.899649 17404 raft_consensus.cc:686] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba [term 1 LEADER]: Becoming Leader. State: Replica: 5aa3fd5c69244b17b0030e272e139dba, State: Running, Role: LEADER
I20260925 05:05:28.900086 17404 consensus_queue.cc:237] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa3fd5c69244b17b0030e272e139dba" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36563 } } peers { permanent_uuid: "1b1cd998c1cc4e06af8af72cd1c8f157" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45619 } } peers { permanent_uuid: "198755afa3c849e583d33e1c5f4ce99a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 46081 } }
I20260925 05:05:28.904789 16932 catalog_manager.cc:5495] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba reported cstate change: term changed from 0 to 1, leader changed from <none> to 5aa3fd5c69244b17b0030e272e139dba (127.30.247.2). New cstate: current_term: 1 leader_uuid: "5aa3fd5c69244b17b0030e272e139dba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa3fd5c69244b17b0030e272e139dba" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 36563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b1cd998c1cc4e06af8af72cd1c8f157" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "198755afa3c849e583d33e1c5f4ce99a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 46081 } health_report { overall_health: UNKNOWN } } }
I20260925 05:05:28.941191 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:28.935428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59186 (local address 127.30.247.2:36563)
0925 05:05:28.935643 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:28.935649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:28.935751 (+ 102us) server_negotiation.cc:407] Connection header received
0925 05:05:28.936011 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:28.936015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:28.936092 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:28.936253 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:05:28.936649 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.937368 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.938821 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.939305 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.939446 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:28.940025 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:05:28.940628 (+ 603us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:05:28.940781 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:28.940840 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 05:05:28.940920 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 05:05:28.949785 17335 raft_consensus.cc:1261] T 06504d4370054d8d9b25752556cb8fbf P 198755afa3c849e583d33e1c5f4ce99a [term 1 FOLLOWER]: Refusing update from remote peer 5aa3fd5c69244b17b0030e272e139dba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:28.949785 17074 raft_consensus.cc:1261] T 06504d4370054d8d9b25752556cb8fbf P 1b1cd998c1cc4e06af8af72cd1c8f157 [term 1 FOLLOWER]: Refusing update from remote peer 5aa3fd5c69244b17b0030e272e139dba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:28.950922 17409 consensus_queue.cc:1035] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b1cd998c1cc4e06af8af72cd1c8f157" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 45619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:28.951297 17404 consensus_queue.cc:1035] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba [LEADER]: Connected to new peer: Peer: permanent_uuid: "198755afa3c849e583d33e1c5f4ce99a" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 46081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:28.964339 17414 mvcc.cc:205] Tried to move back new op lower bound from 7333120937766748160 to 7333120937578807296. Current Snapshot: MvccSnapshot[applied={T|T < 7333120937766748160}]
I20260925 05:05:28.976984 16932 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:28.977201 16932 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:28.980635 16932 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fec928afb1e74d61b1ed3857f53999ba: Sending DeleteTablet for 3 replicas of tablet 06504d4370054d8d9b25752556cb8fbf
I20260925 05:05:28.981629 17185 tablet_service.cc:1514] Processing DeleteTablet for tablet 06504d4370054d8d9b25752556cb8fbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:28 UTC) from {username='slave'} at 127.0.0.1:59176
I20260925 05:05:28.981775 17054 tablet_service.cc:1514] Processing DeleteTablet for tablet 06504d4370054d8d9b25752556cb8fbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:28 UTC) from {username='slave'} at 127.0.0.1:43430
I20260925 05:05:28.981988 17315 tablet_service.cc:1514] Processing DeleteTablet for tablet 06504d4370054d8d9b25752556cb8fbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:28 UTC) from {username='slave'} at 127.0.0.1:41602
I20260925 05:05:28.982517 17421 tablet_replica.cc:330] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba: stopping tablet replica
I20260925 05:05:28.983027 17421 raft_consensus.cc:2229] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:05:28.983319 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 16992
I20260925 05:05:28.983796 17421 raft_consensus.cc:2258] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:28.989758 17421 ts_tablet_manager.cc:1905] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:28.992995 17423 tablet_replica.cc:330] T 06504d4370054d8d9b25752556cb8fbf P 198755afa3c849e583d33e1c5f4ce99a: stopping tablet replica
I20260925 05:05:28.993415 17423 raft_consensus.cc:2229] T 06504d4370054d8d9b25752556cb8fbf P 198755afa3c849e583d33e1c5f4ce99a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:28.993907 17423 raft_consensus.cc:2258] T 06504d4370054d8d9b25752556cb8fbf P 198755afa3c849e583d33e1c5f4ce99a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:28.994609 17422 tablet_replica.cc:330] T 06504d4370054d8d9b25752556cb8fbf P 1b1cd998c1cc4e06af8af72cd1c8f157: stopping tablet replica
I20260925 05:05:28.994976 17422 raft_consensus.cc:2229] T 06504d4370054d8d9b25752556cb8fbf P 1b1cd998c1cc4e06af8af72cd1c8f157 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:28.995446 17422 raft_consensus.cc:2258] T 06504d4370054d8d9b25752556cb8fbf P 1b1cd998c1cc4e06af8af72cd1c8f157 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:28.997459 17422 ts_tablet_manager.cc:1905] T 06504d4370054d8d9b25752556cb8fbf P 1b1cd998c1cc4e06af8af72cd1c8f157: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:28.998922 17421 ts_tablet_manager.cc:1918] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:05:28.999137 17421 log.cc:1196] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-1/wal/wals/06504d4370054d8d9b25752556cb8fbf
I20260925 05:05:28.999740 17421 ts_tablet_manager.cc:1939] T 06504d4370054d8d9b25752556cb8fbf P 5aa3fd5c69244b17b0030e272e139dba: Deleting consensus metadata
I20260925 05:05:29.000957 17423 ts_tablet_manager.cc:1905] T 06504d4370054d8d9b25752556cb8fbf P 198755afa3c849e583d33e1c5f4ce99a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:29.001569 16918 catalog_manager.cc:4841] TS 5aa3fd5c69244b17b0030e272e139dba (127.30.247.2:36563): tablet 06504d4370054d8d9b25752556cb8fbf (table test-table [id=fb7114f7513449b79591a93a3b161cfa]) successfully deleted
I20260925 05:05:29.003492 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:28.985599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43434 (local address 127.30.247.1:45619)
0925 05:05:28.985831 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:28.985840 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:28.995331 (+ 9491us) server_negotiation.cc:407] Connection header received
0925 05:05:28.995764 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:28.995770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:28.995888 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:28.995970 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:28.996640 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:28.997714 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:28.999656 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:29.000311 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:29.000398 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:29.001811 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:29.001821 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:29.001836 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:29.002562 (+ 726us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:29.002622 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:29.002646 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:29.002739 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:29.002974 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 05:05:29.003081 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 05:05:29.004794 17114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:29.009419 17423 ts_tablet_manager.cc:1918] T 06504d4370054d8d9b25752556cb8fbf P 198755afa3c849e583d33e1c5f4ce99a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:05:29.009568 17423 log.cc:1196] T 06504d4370054d8d9b25752556cb8fbf P 198755afa3c849e583d33e1c5f4ce99a: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-2/wal/wals/06504d4370054d8d9b25752556cb8fbf
I20260925 05:05:29.010093 17423 ts_tablet_manager.cc:1939] T 06504d4370054d8d9b25752556cb8fbf P 198755afa3c849e583d33e1c5f4ce99a: Deleting consensus metadata
I20260925 05:05:29.011721 16917 catalog_manager.cc:4841] TS 198755afa3c849e583d33e1c5f4ce99a (127.30.247.3:46081): tablet 06504d4370054d8d9b25752556cb8fbf (table test-table [id=fb7114f7513449b79591a93a3b161cfa]) successfully deleted
I20260925 05:05:29.099668 17422 ts_tablet_manager.cc:1918] T 06504d4370054d8d9b25752556cb8fbf P 1b1cd998c1cc4e06af8af72cd1c8f157: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:05:29.099946 17422 log.cc:1196] T 06504d4370054d8d9b25752556cb8fbf P 1b1cd998c1cc4e06af8af72cd1c8f157: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312430481417-31708-0/minicluster-data/ts-0/wal/wals/06504d4370054d8d9b25752556cb8fbf
I20260925 05:05:29.100219 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 16992
I20260925 05:05:29.117543 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 17123
W20260925 05:05:29.117661 16918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.247.1:45619 (error 108)
I20260925 05:05:29.120129 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:29.119668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:45619 (local address 127.0.0.1:43436)
0925 05:05:29.119853 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:29.119967 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.1:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 05:05:29.120476 16918 catalog_manager.cc:4584] TS 1b1cd998c1cc4e06af8af72cd1c8f157 (127.30.247.1:45619): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 06504d4370054d8d9b25752556cb8fbf: Network error: Client connection negotiation failed: client connection to 127.30.247.1:45619: connect: Connection refused (error 111)
I20260925 05:05:29.124724 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:29.118396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59200 (local address 127.30.247.2:36563)
0925 05:05:29.118584 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:29.118590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:29.118611 (+ 21us) server_negotiation.cc:407] Connection header received
0925 05:05:29.118776 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:29.118783 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:29.118897 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:29.119006 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:29.119477 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:29.120453 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:29.121739 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:29.122168 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:29.122304 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:29.123356 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:29.123366 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:29.123378 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:29.123948 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:29.123992 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:29.124011 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:29.124126 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:29.124262 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 05:05:29.124351 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 05:05:29.125411 17245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:29.174111 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:29.173667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:45619 (local address 127.0.0.1:43448)
0925 05:05:29.173876 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:29.173965 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.1:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 05:05:29.175074 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:29.174741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:45619 (local address 127.0.0.1:43450)
0925 05:05:29.174882 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:29.174955 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.1:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 05:05:29.233394 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 17123
I20260925 05:05:29.235319 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:29.234883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:45619 (local address 127.0.0.1:43452)
0925 05:05:29.235080 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:29.235155 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.1:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 05:05:29.236214 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:29.235900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:45619 (local address 127.0.0.1:43458)
0925 05:05:29.236036 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:29.236106 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.1:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 05:05:29.251899 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 17253
I20260925 05:05:29.258277 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:29.252605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41610 (local address 127.30.247.3:46081)
0925 05:05:29.252789 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:29.252794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:29.252811 (+ 17us) server_negotiation.cc:407] Connection header received
0925 05:05:29.252910 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:29.252914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:29.252994 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:29.253084 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:29.253432 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:29.254127 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:29.255375 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:29.255753 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:29.255852 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:29.256980 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:29.256988 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:29.257002 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:29.257550 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:29.257596 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:29.257615 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:29.257708 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:29.257865 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 05:05:29.257948 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 05:05:29.258925 17375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:29.317029 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:29.316457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:45619 (local address 127.0.0.1:43464)
0925 05:05:29.316719 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:29.316875 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.1:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 05:05:29.318143 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:29.317745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:45619 (local address 127.0.0.1:43472)
0925 05:05:29.317946 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:29.318018 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.1:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260925 05:05:29.347044 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 17253
I20260925 05:05:29.365283 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 16902
I20260925 05:05:29.366400 16963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:29.472069 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 16902
2026-09-25T05:05:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2683 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:05:29 dist-test-slave-5n86 krb5kdc[17432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:05:29 dist-test-slave-5n86 krb5kdc[17432](info): set up 2 sockets
Sep 25 05:05:29 dist-test-slave-5n86 krb5kdc[17432](info): commencing operation
krb5kdc: starting...
W20260925 05:05:30.576916 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:05:30 dist-test-slave-5n86 krb5kdc[17432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:05:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:05:30.808077 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:45369
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:45369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:31.125228 17445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:31.125476 17445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:31.125653 17445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:31.132671 17445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:31.132766 17445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:31.132850 17445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:31.132937 17445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:31.133010 17445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:31.142468 17445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:45369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:45369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:31.143802 17445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:31.145058 17445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:31.156567 17450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:31.156508 17453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:31.156605 17451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:31.157820 17445 server_base.cc:1034] running on GCE node
I20260925 05:05:31.158566 17445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:31.159933 17445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:31.161156 17445 hybrid_clock.cc:648] HybridClock initialized: now 1790312731161110 us; error 51 us; skew 500 ppm
I20260925 05:05:31.161495 17445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:05:31 dist-test-slave-5n86 krb5kdc[17432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312731, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:31.164579 17445 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:05:31.166051 17445 webserver.cc:460] Webserver started at http://127.30.247.62:40041/ using document root <none> and password file <none>
I20260925 05:05:31.166683 17445 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:31.166781 17445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:31.167027 17445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:31.169996 17445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "3f21806a8d284ff699146a1f2068aa8b"
format_stamp: "Formatted at 2026-09-25 05:05:31 on dist-test-slave-5n86"
server_key: "4329f4b81f9f1806ca1eac4c69180e7d"
server_key_iv: "b44f5d326f8b2c25e40e95db2a5507ea"
server_key_version: "encryptionkey@0"
I20260925 05:05:31.170702 17445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "3f21806a8d284ff699146a1f2068aa8b"
format_stamp: "Formatted at 2026-09-25 05:05:31 on dist-test-slave-5n86"
server_key: "4329f4b81f9f1806ca1eac4c69180e7d"
server_key_iv: "b44f5d326f8b2c25e40e95db2a5507ea"
server_key_version: "encryptionkey@0"
I20260925 05:05:31.175670 17445 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260925 05:05:31.179003 17460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:31.180308 17445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:05:31.180455 17445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "3f21806a8d284ff699146a1f2068aa8b"
format_stamp: "Formatted at 2026-09-25 05:05:31 on dist-test-slave-5n86"
server_key: "4329f4b81f9f1806ca1eac4c69180e7d"
server_key_iv: "b44f5d326f8b2c25e40e95db2a5507ea"
server_key_version: "encryptionkey@0"
I20260925 05:05:31.180680 17445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:31.192932 17445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:31.197109 17445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:31.197327 17445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:31.215436 17445 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:45369
I20260925 05:05:31.215456 17511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:45369 every 8 connection(s)
I20260925 05:05:31.216799 17445 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:31.218547 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 17445
I20260925 05:05:31.218729 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:05:31.219272 31708 external_mini_cluster.cc:1428] Setting key 6903de9235b5322ce034866643322457
I20260925 05:05:31.221302 17512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 05:05:31 dist-test-slave-5n86 krb5kdc[17432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:05:31.234346 17512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f21806a8d284ff699146a1f2068aa8b: Bootstrap starting.
I20260925 05:05:31.238435 17512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f21806a8d284ff699146a1f2068aa8b: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:31.239632 17512 log.cc:824] T 00000000000000000000000000000000 P 3f21806a8d284ff699146a1f2068aa8b: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:31.242905 17512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f21806a8d284ff699146a1f2068aa8b: No bootstrap required, opened a new log
I20260925 05:05:31.243263 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:31.221113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49220 (local address 127.30.247.62:45369)
0925 05:05:31.221933 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:31.221947 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:31.221994 (+ 47us) server_negotiation.cc:407] Connection header received
0925 05:05:31.224178 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:31.224211 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:31.224717 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:31.225136 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:31.226346 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:31.228362 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:31.229670 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:31.230696 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:31.230888 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:31.234945 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:31.234954 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:31.234971 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:31.235017 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:31.238969 (+ 3952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:31.239666 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:31.239675 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:31.239682 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:31.239883 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:31.240297 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:31.240302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:31.240311 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:31.240899 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:31.241188 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:31.241577 (+ 389us) server_negotiation.cc:299] Negotiation successful
0925 05:05:31.241993 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":384,"threads_started":1}
I20260925 05:05:31.250891 17512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f21806a8d284ff699146a1f2068aa8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f21806a8d284ff699146a1f2068aa8b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45369 } }
I20260925 05:05:31.251206 17512 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f21806a8d284ff699146a1f2068aa8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:31.251276 17512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f21806a8d284ff699146a1f2068aa8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f21806a8d284ff699146a1f2068aa8b, State: Initialized, Role: FOLLOWER
I20260925 05:05:31.251804 17512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f21806a8d284ff699146a1f2068aa8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f21806a8d284ff699146a1f2068aa8b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45369 } }
I20260925 05:05:31.252005 17512 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f21806a8d284ff699146a1f2068aa8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:31.252099 17512 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f21806a8d284ff699146a1f2068aa8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:31.252218 17512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f21806a8d284ff699146a1f2068aa8b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:31.254241 17512 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f21806a8d284ff699146a1f2068aa8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f21806a8d284ff699146a1f2068aa8b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45369 } }
I20260925 05:05:31.254644 17512 leader_election.cc:302] T 00000000000000000000000000000000 P 3f21806a8d284ff699146a1f2068aa8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f21806a8d284ff699146a1f2068aa8b; no voters:
I20260925 05:05:31.255298 17512 leader_election.cc:288] T 00000000000000000000000000000000 P 3f21806a8d284ff699146a1f2068aa8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:31.255419 17517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f21806a8d284ff699146a1f2068aa8b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:31.256081 17517 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f21806a8d284ff699146a1f2068aa8b [term 1 LEADER]: Becoming Leader. State: Replica: 3f21806a8d284ff699146a1f2068aa8b, State: Running, Role: LEADER
I20260925 05:05:31.256675 17517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f21806a8d284ff699146a1f2068aa8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f21806a8d284ff699146a1f2068aa8b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45369 } }
I20260925 05:05:31.257414 17512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f21806a8d284ff699146a1f2068aa8b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:31.259312 17519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f21806a8d284ff699146a1f2068aa8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f21806a8d284ff699146a1f2068aa8b. Latest consensus state: current_term: 1 leader_uuid: "3f21806a8d284ff699146a1f2068aa8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f21806a8d284ff699146a1f2068aa8b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45369 } } }
I20260925 05:05:31.259693 17519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f21806a8d284ff699146a1f2068aa8b [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:31.259287 17518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f21806a8d284ff699146a1f2068aa8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f21806a8d284ff699146a1f2068aa8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f21806a8d284ff699146a1f2068aa8b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45369 } } }
I20260925 05:05:31.259861 17518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f21806a8d284ff699146a1f2068aa8b [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:31.262717 17523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:31.267517 17523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:31.276192 17523 catalog_manager.cc:1347] Generated new cluster ID: 573fdef0e77b4729bc8f843e144db5f6
I20260925 05:05:31.276288 17523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:05:31.285584 17523 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:31.287261 17523 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:31.299095 17523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f21806a8d284ff699146a1f2068aa8b: Generated new TSK 0
I20260925 05:05:31.300000 17523 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
I20260925 05:05:31.356739 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:0
--local_ip_for_outbound_sockets=127.30.247.1
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:45369
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:31.674341 17538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:31.674702 17538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:31.674937 17538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:31.682652 17538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:31.682785 17538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:31.682943 17538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:05:31.692438 17538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:45369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:31.693619 17538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:31.694852 17538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:31.707557 17546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:31.707592 17544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:31.707947 17543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:31.708252 17538 server_base.cc:1034] running on GCE node
I20260925 05:05:31.708920 17538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:31.709757 17538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:31.710965 17538 hybrid_clock.cc:648] HybridClock initialized: now 1790312731710923 us; error 52 us; skew 500 ppm
I20260925 05:05:31.711248 17538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:05:31 dist-test-slave-5n86 krb5kdc[17432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312731, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:31.714174 17538 init.cc:377] Logged in from keytab as kudu/127.30.247.1@KRBTEST.COM (short username kudu)
I20260925 05:05:31.715548 17538 webserver.cc:460] Webserver started at http://127.30.247.1:41991/ using document root <none> and password file <none>
I20260925 05:05:31.716174 17538 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:31.716311 17538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:31.716663 17538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:31.719347 17538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-0/data/instance:
uuid: "b2153ae384b54a2db238b23b788630ea"
format_stamp: "Formatted at 2026-09-25 05:05:31 on dist-test-slave-5n86"
server_key: "fdaca01ea39323ef43bee86fc15a2edf"
server_key_iv: "b17c59096323647de532acae28fd6647"
server_key_version: "encryptionkey@0"
I20260925 05:05:31.720064 17538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance:
uuid: "b2153ae384b54a2db238b23b788630ea"
format_stamp: "Formatted at 2026-09-25 05:05:31 on dist-test-slave-5n86"
server_key: "fdaca01ea39323ef43bee86fc15a2edf"
server_key_iv: "b17c59096323647de532acae28fd6647"
server_key_version: "encryptionkey@0"
I20260925 05:05:31.724848 17538 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260925 05:05:31.728012 17553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:31.729298 17538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:05:31.729463 17538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "b2153ae384b54a2db238b23b788630ea"
format_stamp: "Formatted at 2026-09-25 05:05:31 on dist-test-slave-5n86"
server_key: "fdaca01ea39323ef43bee86fc15a2edf"
server_key_iv: "b17c59096323647de532acae28fd6647"
server_key_version: "encryptionkey@0"
I20260925 05:05:31.729656 17538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:31.743045 17538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:31.746744 17538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:31.747022 17538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:31.747885 17538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:31.749147 17538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:31.749243 17538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:31.749380 17538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:31.749450 17538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:31.781038 17538 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:37291
I20260925 05:05:31.781088 17665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:37291 every 8 connection(s)
I20260925 05:05:31.782287 17538 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:05:31.789052 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 17538
I20260925 05:05:31.789250 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance
I20260925 05:05:31.789682 31708 external_mini_cluster.cc:1428] Setting key d7868a3489b909c56994c245eb7004f5
Sep 25 05:05:31 dist-test-slave-5n86 krb5kdc[17432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312731, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:05:31.805794 17668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:31.784932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:35699 (local address 127.30.247.62:45369)
0925 05:05:31.785356 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:31.785361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:31.786340 (+ 979us) server_negotiation.cc:407] Connection header received
0925 05:05:31.788130 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:31.788136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:31.788226 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:31.788347 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:31.790087 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:31.791091 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:31.793501 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:31.793920 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:31.794025 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:31.798975 (+ 4950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:31.798980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:31.798983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:31.799018 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:31.802284 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:31.803032 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:31.803041 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:31.803047 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:31.803214 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:31.803642 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:31.803650 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:31.803656 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:31.804046 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:31.804209 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:31.805034 (+ 825us) server_negotiation.cc:299] Negotiation successful
0925 05:05:31.805311 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":219,"threads_started":1}
I20260925 05:05:31.806433 17669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:31.785510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:45369 (local address 127.30.247.1:35699)
0925 05:05:31.786177 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:31.786216 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:31.787870 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:31.788705 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:31.788714 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:31.789285 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:31.789916 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:31.789935 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:31.791294 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:31.791301 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:31.793346 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:31.793365 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:31.794144 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:31.794154 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:31.794481 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:31.795485 (+ 1004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:31.795519 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:31.798733 (+ 3214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:31.802435 (+ 3702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:31.802442 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:31.802455 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:31.802898 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:31.803329 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:31.803332 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:31.803335 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:31.803525 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:31.804205 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:31.804211 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:31.804623 (+ 412us) client_negotiation.cc:769] Sending connection context
0925 05:05:31.804895 (+ 272us) client_negotiation.cc:240] Negotiation successful
0925 05:05:31.805289 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":258,"threads_started":1}
I20260925 05:05:31.807992 17666 heartbeater.cc:347] Connected to a master server at 127.30.247.62:45369
I20260925 05:05:31.808329 17666 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:31.809266 17666 heartbeater.cc:511] Master 127.30.247.62:45369 requested a full tablet report, sending...
I20260925 05:05:31.811457 17477 ts_manager.cc:194] Registered new tserver with Master: b2153ae384b54a2db238b23b788630ea (127.30.247.1:37291)
I20260925 05:05:31.813078 17477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.1@KRBTEST.COM'} at 127.30.247.1:35699
WARNING: no policy specified for kudu/127.30.247.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
I20260925 05:05:31.851725 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:0
--local_ip_for_outbound_sockets=127.30.247.2
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:45369
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:32.172916 17672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:32.173226 17672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:32.173390 17672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:32.180415 17672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:32.180575 17672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:32.180758 17672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:05:32.190251 17672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:45369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:32.191470 17672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:32.192836 17672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:32.205682 17677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:32.205682 17678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:32.205924 17680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:32.206614 17672 server_base.cc:1034] running on GCE node
I20260925 05:05:32.207118 17672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:32.207916 17672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:32.209125 17672 hybrid_clock.cc:648] HybridClock initialized: now 1790312732209084 us; error 25 us; skew 500 ppm
I20260925 05:05:32.209451 17672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:05:32 dist-test-slave-5n86 krb5kdc[17432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312732, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:32.212467 17672 init.cc:377] Logged in from keytab as kudu/127.30.247.2@KRBTEST.COM (short username kudu)
I20260925 05:05:32.213783 17672 webserver.cc:460] Webserver started at http://127.30.247.2:38555/ using document root <none> and password file <none>
I20260925 05:05:32.214372 17672 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:32.214468 17672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:32.214710 17672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:32.217512 17672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-1/data/instance:
uuid: "b3ba84eec8b7481b9dbf833fdcc24c96"
format_stamp: "Formatted at 2026-09-25 05:05:32 on dist-test-slave-5n86"
server_key: "dcfabb52be14eef53d5823834566db57"
server_key_iv: "e7775276bba12c015a935d9d2f9ccc19"
server_key_version: "encryptionkey@0"
I20260925 05:05:32.218232 17672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance:
uuid: "b3ba84eec8b7481b9dbf833fdcc24c96"
format_stamp: "Formatted at 2026-09-25 05:05:32 on dist-test-slave-5n86"
server_key: "dcfabb52be14eef53d5823834566db57"
server_key_iv: "e7775276bba12c015a935d9d2f9ccc19"
server_key_version: "encryptionkey@0"
I20260925 05:05:32.223131 17672 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 05:05:32.226629 17687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:32.228027 17672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:05:32.228194 17672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "b3ba84eec8b7481b9dbf833fdcc24c96"
format_stamp: "Formatted at 2026-09-25 05:05:32 on dist-test-slave-5n86"
server_key: "dcfabb52be14eef53d5823834566db57"
server_key_iv: "e7775276bba12c015a935d9d2f9ccc19"
server_key_version: "encryptionkey@0"
I20260925 05:05:32.228387 17672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:32.252373 17672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:32.256407 17672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:32.256711 17672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:32.257627 17672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:32.258922 17672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:32.259016 17672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:32.259150 17672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:32.259217 17672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:32.291394 17672 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:39763
I20260925 05:05:32.291440 17799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:39763 every 8 connection(s)
I20260925 05:05:32.292716 17672 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:05:32.294658 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 17672
I20260925 05:05:32.294872 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance
I20260925 05:05:32.295442 31708 external_mini_cluster.cc:1428] Setting key f6d09178943ec4df177209a96f4cf17d
Sep 25 05:05:32 dist-test-slave-5n86 krb5kdc[17432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312732, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:05:32.312126 17668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:32.295275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:33295 (local address 127.30.247.62:45369)
0925 05:05:32.295480 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:32.295487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:32.296735 (+ 1248us) server_negotiation.cc:407] Connection header received
0925 05:05:32.298531 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:32.298539 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:32.298627 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:32.298732 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:32.300163 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.301303 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.302785 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.303180 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.303334 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:32.306573 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:32.306577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:32.306581 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:32.306612 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:32.309308 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:32.309970 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:32.309979 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:32.309983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:32.310124 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:32.310524 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:32.310529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:32.310533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:32.310775 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:32.310925 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:32.311554 (+ 629us) server_negotiation.cc:299] Negotiation successful
0925 05:05:32.311792 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 05:05:32.313102 17802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:32.295762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:45369 (local address 127.30.247.2:33295)
0925 05:05:32.296574 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:32.296624 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:32.298264 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:32.298952 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:32.298959 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:32.299460 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:32.299997 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:32.300014 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.301442 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.301446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:32.302639 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:32.302651 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.303395 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.303405 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:32.303632 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:32.304276 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:32.304303 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:32.306374 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:32.309439 (+ 3065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:32.309448 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:32.309462 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:32.309857 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:32.310231 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:32.310236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:32.310240 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:32.310415 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:32.310898 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:32.310905 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:32.311209 (+ 304us) client_negotiation.cc:769] Sending connection context
0925 05:05:32.311460 (+ 251us) client_negotiation.cc:240] Negotiation successful
0925 05:05:32.311968 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":330,"threads_started":1}
I20260925 05:05:32.314503 17800 heartbeater.cc:347] Connected to a master server at 127.30.247.62:45369
I20260925 05:05:32.314785 17800 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:32.315469 17800 heartbeater.cc:511] Master 127.30.247.62:45369 requested a full tablet report, sending...
I20260925 05:05:32.316969 17477 ts_manager.cc:194] Registered new tserver with Master: b3ba84eec8b7481b9dbf833fdcc24c96 (127.30.247.2:39763)
I20260925 05:05:32.318185 17477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.2@KRBTEST.COM'} at 127.30.247.2:33295
WARNING: no policy specified for kudu/127.30.247.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
I20260925 05:05:32.352010 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:0
--local_ip_for_outbound_sockets=127.30.247.3
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:45369
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:32.661324 17805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:32.661576 17805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:32.661664 17805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:32.668593 17805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:32.668686 17805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:32.668790 17805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:05:32.677906 17805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:45369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:32.679018 17805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:32.680198 17805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:32.692920 17811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:32.692992 17810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:32.692965 17813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:32.694145 17805 server_base.cc:1034] running on GCE node
I20260925 05:05:32.694696 17805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:32.695501 17805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:32.696691 17805 hybrid_clock.cc:648] HybridClock initialized: now 1790312732696634 us; error 50 us; skew 500 ppm
I20260925 05:05:32.696949 17805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:05:32 dist-test-slave-5n86 krb5kdc[17432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312732, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:32.700130 17805 init.cc:377] Logged in from keytab as kudu/127.30.247.3@KRBTEST.COM (short username kudu)
I20260925 05:05:32.701560 17805 webserver.cc:460] Webserver started at http://127.30.247.3:37075/ using document root <none> and password file <none>
I20260925 05:05:32.702190 17805 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:32.702316 17805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:32.702623 17805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:32.705545 17805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-2/data/instance:
uuid: "ecea1053db6a46568968671e41dcb804"
format_stamp: "Formatted at 2026-09-25 05:05:32 on dist-test-slave-5n86"
server_key: "b0070371cd8e802d736053c76d7d4664"
server_key_iv: "17249e4e22840a36fa1b200a2c01e85d"
server_key_version: "encryptionkey@0"
I20260925 05:05:32.706354 17805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance:
uuid: "ecea1053db6a46568968671e41dcb804"
format_stamp: "Formatted at 2026-09-25 05:05:32 on dist-test-slave-5n86"
server_key: "b0070371cd8e802d736053c76d7d4664"
server_key_iv: "17249e4e22840a36fa1b200a2c01e85d"
server_key_version: "encryptionkey@0"
I20260925 05:05:32.711410 17805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 05:05:32.714927 17820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:32.716212 17805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:05:32.716373 17805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "ecea1053db6a46568968671e41dcb804"
format_stamp: "Formatted at 2026-09-25 05:05:32 on dist-test-slave-5n86"
server_key: "b0070371cd8e802d736053c76d7d4664"
server_key_iv: "17249e4e22840a36fa1b200a2c01e85d"
server_key_version: "encryptionkey@0"
I20260925 05:05:32.716614 17805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:32.730850 17805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:32.735582 17805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:32.735857 17805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:32.736886 17805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:32.738216 17805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:32.738291 17805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:32.738456 17805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:32.738511 17805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:32.771361 17805 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:35475
I20260925 05:05:32.771445 17932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:35475 every 8 connection(s)
I20260925 05:05:32.772686 17805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:05:32.782878 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 17805
I20260925 05:05:32.783082 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance
I20260925 05:05:32.783571 31708 external_mini_cluster.cc:1428] Setting key 9a2d295be7a4aa07594a79ed47576c4e
Sep 25 05:05:32 dist-test-slave-5n86 krb5kdc[17432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312732, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:05:32.793489 17668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:32.775247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:56459 (local address 127.30.247.62:45369)
0925 05:05:32.775465 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:32.775472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:32.776805 (+ 1333us) server_negotiation.cc:407] Connection header received
0925 05:05:32.778566 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:32.778570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:32.778638 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:32.778731 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:32.780111 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.781300 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.782892 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.783559 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.783676 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:32.788273 (+ 4597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:32.788277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:32.788280 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:32.788310 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:32.790794 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:32.791493 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:32.791498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:32.791500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:32.791632 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:32.792056 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:32.792059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:32.792061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:32.792318 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:32.792426 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:32.793094 (+ 668us) server_negotiation.cc:299] Negotiation successful
0925 05:05:32.793258 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 05:05:32.794399 17935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:32.775729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:45369 (local address 127.30.247.3:56459)
0925 05:05:32.776653 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:32.776690 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:32.778296 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:32.778936 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:32.778943 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:32.779415 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:32.779944 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:32.779960 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.781446 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.781450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:32.782736 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:32.782749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.784007 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.784014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:32.784277 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:32.785242 (+ 965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:32.785276 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:32.788079 (+ 2803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:32.790947 (+ 2868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:32.790953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:32.790962 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:32.791353 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:32.791771 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:32.791775 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:32.791778 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:32.791956 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:32.792438 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:32.792444 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:32.792781 (+ 337us) client_negotiation.cc:769] Sending connection context
0925 05:05:32.793002 (+ 221us) client_negotiation.cc:240] Negotiation successful
0925 05:05:32.793339 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":415,"threads_started":1}
I20260925 05:05:32.795702 17933 heartbeater.cc:347] Connected to a master server at 127.30.247.62:45369
I20260925 05:05:32.795964 17933 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:32.796676 17933 heartbeater.cc:511] Master 127.30.247.62:45369 requested a full tablet report, sending...
I20260925 05:05:32.798426 17477 ts_manager.cc:194] Registered new tserver with Master: ecea1053db6a46568968671e41dcb804 (127.30.247.3:35475)
I20260925 05:05:32.799388 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:05:32.799710 17477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.3@KRBTEST.COM'} at 127.30.247.3:56459
I20260925 05:05:32.815397 17668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:32.803676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49224 (local address 127.30.247.62:45369)
0925 05:05:32.803822 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:32.803826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:32.804249 (+ 423us) server_negotiation.cc:407] Connection header received
0925 05:05:32.804363 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:32.804367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:32.804419 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:32.804498 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:32.805790 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.806454 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.807736 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.808136 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.808222 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:32.810650 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:32.810654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:32.810658 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:32.810685 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:32.813176 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:32.813778 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:32.813781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:32.813783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:32.813909 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:32.814335 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:32.814338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:32.814340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:32.814601 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:32.814780 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:32.815080 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 05:05:32.815195 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 05:05:32.816713 17666 heartbeater.cc:503] Master 127.30.247.62:45369 was elected leader, sending a full tablet report...
I20260925 05:05:32.818953 17477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:05:32.825384 17477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:05:32.846868 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:32.836450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:35475 (local address 127.0.0.1:33520)
0925 05:05:32.836995 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:32.837012 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:32.837127 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:32.838760 (+ 1633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:32.838770 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:32.838826 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:32.839869 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:32.839881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.841825 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.841830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:32.844440 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:32.844451 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.845846 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.845861 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:32.846055 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:32.846107 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 05:05:32.846224 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 05:05:32.846414 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":129,"thread_start_us":2289,"threads_started":1}
I20260925 05:05:32.847378 17945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:32.836032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:39763 (local address 127.0.0.1:37962)
0925 05:05:32.836859 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:32.836914 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:32.837127 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:32.840284 (+ 3157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:32.840289 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:32.840315 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:32.840704 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:32.840717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.842567 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.842573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:32.844789 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:32.844799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.846309 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.846315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:32.846473 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:32.846505 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 05:05:32.846578 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 05:05:32.846696 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":89,"thread_start_us":2549,"threads_started":1}
I20260925 05:05:32.847757 17946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:32.836211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37962 (local address 127.30.247.2:39763)
0925 05:05:32.839466 (+ 3255us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:32.839706 (+ 240us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:32.839737 (+ 31us) server_negotiation.cc:407] Connection header received
0925 05:05:32.839850 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:32.839860 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:32.840087 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:32.840280 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:32.840919 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.842404 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.844930 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.846079 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.846950 (+ 871us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:32.847158 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:32.847250 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 05:05:32.847434 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3115,"thread_start_us":3015,"threads_started":1}
I20260925 05:05:32.848088 17949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:32.837444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33520 (local address 127.30.247.3:35475)
0925 05:05:32.838054 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:32.838064 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:32.838094 (+ 30us) server_negotiation.cc:407] Connection header received
0925 05:05:32.838186 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:32.838197 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:32.838484 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:32.839497 (+ 1013us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:32.840069 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.841674 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.844611 (+ 2937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.845642 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.845778 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:32.845996 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:32.847576 (+ 1580us) server_negotiation.cc:299] Negotiation successful
0925 05:05:32.847721 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":601,"threads_started":1}
I20260925 05:05:32.848155 17948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:32.837055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:37291 (local address 127.0.0.1:47360)
0925 05:05:32.837742 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:32.837762 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:32.837881 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:32.840704 (+ 2823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:32.840708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:32.840733 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:32.841066 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:32.841075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.842686 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.842691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:32.844431 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:32.844445 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.847390 (+ 2945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.847400 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:32.847585 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:32.847627 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 05:05:32.847697 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 05:05:32.847827 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":87,"thread_start_us":976,"threads_started":1}
I20260925 05:05:32.849146 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:32.837940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47360 (local address 127.30.247.1:37291)
0925 05:05:32.840148 (+ 2208us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:32.840156 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:32.840182 (+ 26us) server_negotiation.cc:407] Connection header received
0925 05:05:32.840268 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:32.840274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:32.840486 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:32.840716 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:32.841206 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.842527 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.846076 (+ 3549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.847148 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.848388 (+ 1240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:32.848633 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:32.848683 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 05:05:32.848841 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2054,"thread_start_us":1919,"threads_started":1}
I20260925 05:05:32.851536 17735 tablet_service.cc:1467] Processing CreateTablet for tablet b241ecb8d98e4f7b80f0a1223bda5118 (DEFAULT_TABLE table=test-table [id=d494212e5dd74c97876936990755f2dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:32.851612 17868 tablet_service.cc:1467] Processing CreateTablet for tablet b241ecb8d98e4f7b80f0a1223bda5118 (DEFAULT_TABLE table=test-table [id=d494212e5dd74c97876936990755f2dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:32.852833 17735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b241ecb8d98e4f7b80f0a1223bda5118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:32.853159 17868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b241ecb8d98e4f7b80f0a1223bda5118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:32.853050 17601 tablet_service.cc:1467] Processing CreateTablet for tablet b241ecb8d98e4f7b80f0a1223bda5118 (DEFAULT_TABLE table=test-table [id=d494212e5dd74c97876936990755f2dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:32.854315 17601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b241ecb8d98e4f7b80f0a1223bda5118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:32.864176 17951 tablet_bootstrap.cc:492] T b241ecb8d98e4f7b80f0a1223bda5118 P b3ba84eec8b7481b9dbf833fdcc24c96: Bootstrap starting.
I20260925 05:05:32.867082 17952 tablet_bootstrap.cc:492] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804: Bootstrap starting.
I20260925 05:05:32.867619 17951 tablet_bootstrap.cc:654] T b241ecb8d98e4f7b80f0a1223bda5118 P b3ba84eec8b7481b9dbf833fdcc24c96: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:32.869266 17953 tablet_bootstrap.cc:492] T b241ecb8d98e4f7b80f0a1223bda5118 P b2153ae384b54a2db238b23b788630ea: Bootstrap starting.
I20260925 05:05:32.869288 17951 log.cc:824] T b241ecb8d98e4f7b80f0a1223bda5118 P b3ba84eec8b7481b9dbf833fdcc24c96: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:32.870443 17952 tablet_bootstrap.cc:654] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:32.871579 17952 log.cc:824] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:32.872131 17953 tablet_bootstrap.cc:654] T b241ecb8d98e4f7b80f0a1223bda5118 P b2153ae384b54a2db238b23b788630ea: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:32.872426 17951 tablet_bootstrap.cc:492] T b241ecb8d98e4f7b80f0a1223bda5118 P b3ba84eec8b7481b9dbf833fdcc24c96: No bootstrap required, opened a new log
I20260925 05:05:32.872707 17951 ts_tablet_manager.cc:1397] T b241ecb8d98e4f7b80f0a1223bda5118 P b3ba84eec8b7481b9dbf833fdcc24c96: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 05:05:32.873291 17953 log.cc:824] T b241ecb8d98e4f7b80f0a1223bda5118 P b2153ae384b54a2db238b23b788630ea: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:32.874662 17952 tablet_bootstrap.cc:492] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804: No bootstrap required, opened a new log
I20260925 05:05:32.874919 17952 ts_tablet_manager.cc:1397] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 05:05:32.875609 17953 tablet_bootstrap.cc:492] T b241ecb8d98e4f7b80f0a1223bda5118 P b2153ae384b54a2db238b23b788630ea: No bootstrap required, opened a new log
I20260925 05:05:32.875855 17953 ts_tablet_manager.cc:1397] T b241ecb8d98e4f7b80f0a1223bda5118 P b2153ae384b54a2db238b23b788630ea: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 05:05:32.882827 17953 raft_consensus.cc:348] T b241ecb8d98e4f7b80f0a1223bda5118 P b2153ae384b54a2db238b23b788630ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ba84eec8b7481b9dbf833fdcc24c96" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 39763 } } peers { permanent_uuid: "ecea1053db6a46568968671e41dcb804" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35475 } } peers { permanent_uuid: "b2153ae384b54a2db238b23b788630ea" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 37291 } }
I20260925 05:05:32.883174 17953 raft_consensus.cc:374] T b241ecb8d98e4f7b80f0a1223bda5118 P b2153ae384b54a2db238b23b788630ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:32.883278 17953 raft_consensus.cc:729] T b241ecb8d98e4f7b80f0a1223bda5118 P b2153ae384b54a2db238b23b788630ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2153ae384b54a2db238b23b788630ea, State: Initialized, Role: FOLLOWER
I20260925 05:05:32.883052 17951 raft_consensus.cc:348] T b241ecb8d98e4f7b80f0a1223bda5118 P b3ba84eec8b7481b9dbf833fdcc24c96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ba84eec8b7481b9dbf833fdcc24c96" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 39763 } } peers { permanent_uuid: "ecea1053db6a46568968671e41dcb804" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35475 } } peers { permanent_uuid: "b2153ae384b54a2db238b23b788630ea" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 37291 } }
I20260925 05:05:32.883498 17951 raft_consensus.cc:374] T b241ecb8d98e4f7b80f0a1223bda5118 P b3ba84eec8b7481b9dbf833fdcc24c96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:32.883648 17951 raft_consensus.cc:729] T b241ecb8d98e4f7b80f0a1223bda5118 P b3ba84eec8b7481b9dbf833fdcc24c96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ba84eec8b7481b9dbf833fdcc24c96, State: Initialized, Role: FOLLOWER
I20260925 05:05:32.883927 17953 consensus_queue.cc:260] T b241ecb8d98e4f7b80f0a1223bda5118 P b2153ae384b54a2db238b23b788630ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ba84eec8b7481b9dbf833fdcc24c96" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 39763 } } peers { permanent_uuid: "ecea1053db6a46568968671e41dcb804" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35475 } } peers { permanent_uuid: "b2153ae384b54a2db238b23b788630ea" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 37291 } }
I20260925 05:05:32.884199 17951 consensus_queue.cc:260] T b241ecb8d98e4f7b80f0a1223bda5118 P b3ba84eec8b7481b9dbf833fdcc24c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ba84eec8b7481b9dbf833fdcc24c96" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 39763 } } peers { permanent_uuid: "ecea1053db6a46568968671e41dcb804" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35475 } } peers { permanent_uuid: "b2153ae384b54a2db238b23b788630ea" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 37291 } }
I20260925 05:05:32.885727 17953 ts_tablet_manager.cc:1428] T b241ecb8d98e4f7b80f0a1223bda5118 P b2153ae384b54a2db238b23b788630ea: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260925 05:05:32.885403 17952 raft_consensus.cc:348] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ba84eec8b7481b9dbf833fdcc24c96" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 39763 } } peers { permanent_uuid: "ecea1053db6a46568968671e41dcb804" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35475 } } peers { permanent_uuid: "b2153ae384b54a2db238b23b788630ea" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 37291 } }
I20260925 05:05:32.885882 17952 raft_consensus.cc:374] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:32.886080 17952 raft_consensus.cc:729] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecea1053db6a46568968671e41dcb804, State: Initialized, Role: FOLLOWER
I20260925 05:05:32.886830 17952 consensus_queue.cc:260] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ba84eec8b7481b9dbf833fdcc24c96" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 39763 } } peers { permanent_uuid: "ecea1053db6a46568968671e41dcb804" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35475 } } peers { permanent_uuid: "b2153ae384b54a2db238b23b788630ea" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 37291 } }
I20260925 05:05:32.888010 17933 heartbeater.cc:503] Master 127.30.247.62:45369 was elected leader, sending a full tablet report...
I20260925 05:05:32.888899 17952 ts_tablet_manager.cc:1428] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260925 05:05:32.890483 17800 heartbeater.cc:503] Master 127.30.247.62:45369 was elected leader, sending a full tablet report...
I20260925 05:05:32.892498 17951 ts_tablet_manager.cc:1428] T b241ecb8d98e4f7b80f0a1223bda5118 P b3ba84eec8b7481b9dbf833fdcc24c96: Time spent starting tablet: real 0.020s user 0.011s sys 0.007s
I20260925 05:05:32.927959 17959 raft_consensus.cc:482] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:05:32.928280 17959 raft_consensus.cc:504] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ba84eec8b7481b9dbf833fdcc24c96" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 39763 } } peers { permanent_uuid: "ecea1053db6a46568968671e41dcb804" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35475 } } peers { permanent_uuid: "b2153ae384b54a2db238b23b788630ea" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 37291 } }
I20260925 05:05:32.930047 17959 leader_election.cc:288] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3ba84eec8b7481b9dbf833fdcc24c96 (127.30.247.2:39763), b2153ae384b54a2db238b23b788630ea (127.30.247.1:37291)
I20260925 05:05:32.937368 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:32.930933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:39183 (local address 127.30.247.1:37291)
0925 05:05:32.931130 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:32.931134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:32.931151 (+ 17us) server_negotiation.cc:407] Connection header received
0925 05:05:32.931209 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:32.931213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:32.931268 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:32.931342 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:32.931839 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.932931 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.935331 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.936560 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.936730 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:32.936816 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:32.936998 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 05:05:32.937123 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 05:05:32.937402 17935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:32.930387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:37291 (local address 127.30.247.3:39183)
0925 05:05:32.930756 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:32.930781 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:32.930932 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:32.931376 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:32.931383 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:32.931410 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:32.931636 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:32.931646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.933043 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.933047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:32.935162 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:32.935177 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.936689 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.936693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:32.936817 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:32.936840 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 05:05:32.936972 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 05:05:32.937104 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":66}
I20260925 05:05:32.937741 17946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:32.930347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:33357 (local address 127.30.247.2:39763)
0925 05:05:32.930559 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:32.930566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:32.931250 (+ 684us) server_negotiation.cc:407] Connection header received
0925 05:05:32.931467 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:32.931476 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:32.931543 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:32.931611 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:32.932221 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.933253 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.935561 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.936801 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.936966 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:32.937033 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:32.937335 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 05:05:32.937487 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 05:05:32.938350 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:32.930413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:39763 (local address 127.30.247.3:33357)
0925 05:05:32.931161 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:32.931174 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:32.931296 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:32.931642 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:32.931646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:32.931670 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:32.932019 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:32.932027 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.933416 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.933422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:32.935413 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:32.935423 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.936990 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.936995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:32.937149 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:32.937179 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 05:05:32.937869 (+ 690us) client_negotiation.cc:240] Negotiation successful
0925 05:05:32.938004 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":1,"thread_start_us":391,"threads_started":1}
I20260925 05:05:32.938707 17621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b241ecb8d98e4f7b80f0a1223bda5118" candidate_uuid: "ecea1053db6a46568968671e41dcb804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2153ae384b54a2db238b23b788630ea" is_pre_election: true
I20260925 05:05:32.939186 17621 raft_consensus.cc:2454] T b241ecb8d98e4f7b80f0a1223bda5118 P b2153ae384b54a2db238b23b788630ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecea1053db6a46568968671e41dcb804 in term 0.
I20260925 05:05:32.939292 17755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b241ecb8d98e4f7b80f0a1223bda5118" candidate_uuid: "ecea1053db6a46568968671e41dcb804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3ba84eec8b7481b9dbf833fdcc24c96" is_pre_election: true
I20260925 05:05:32.939857 17755 raft_consensus.cc:2454] T b241ecb8d98e4f7b80f0a1223bda5118 P b3ba84eec8b7481b9dbf833fdcc24c96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecea1053db6a46568968671e41dcb804 in term 0.
I20260925 05:05:32.939929 17824 leader_election.cc:302] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2153ae384b54a2db238b23b788630ea, ecea1053db6a46568968671e41dcb804; no voters:
I20260925 05:05:32.940326 17959 raft_consensus.cc:2781] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:05:32.940474 17959 raft_consensus.cc:482] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:05:32.940599 17959 raft_consensus.cc:3037] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:32.942729 17959 raft_consensus.cc:504] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ba84eec8b7481b9dbf833fdcc24c96" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 39763 } } peers { permanent_uuid: "ecea1053db6a46568968671e41dcb804" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35475 } } peers { permanent_uuid: "b2153ae384b54a2db238b23b788630ea" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 37291 } }
I20260925 05:05:32.943343 17959 leader_election.cc:288] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804 [CANDIDATE]: Term 1 election: Requested vote from peers b3ba84eec8b7481b9dbf833fdcc24c96 (127.30.247.2:39763), b2153ae384b54a2db238b23b788630ea (127.30.247.1:37291)
I20260925 05:05:32.943696 17755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b241ecb8d98e4f7b80f0a1223bda5118" candidate_uuid: "ecea1053db6a46568968671e41dcb804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3ba84eec8b7481b9dbf833fdcc24c96"
I20260925 05:05:32.943760 17621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b241ecb8d98e4f7b80f0a1223bda5118" candidate_uuid: "ecea1053db6a46568968671e41dcb804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2153ae384b54a2db238b23b788630ea"
I20260925 05:05:32.943910 17755 raft_consensus.cc:3037] T b241ecb8d98e4f7b80f0a1223bda5118 P b3ba84eec8b7481b9dbf833fdcc24c96 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:32.943930 17621 raft_consensus.cc:3037] T b241ecb8d98e4f7b80f0a1223bda5118 P b2153ae384b54a2db238b23b788630ea [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:32.947065 17621 raft_consensus.cc:2454] T b241ecb8d98e4f7b80f0a1223bda5118 P b2153ae384b54a2db238b23b788630ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecea1053db6a46568968671e41dcb804 in term 1.
I20260925 05:05:32.947065 17755 raft_consensus.cc:2454] T b241ecb8d98e4f7b80f0a1223bda5118 P b3ba84eec8b7481b9dbf833fdcc24c96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecea1053db6a46568968671e41dcb804 in term 1.
I20260925 05:05:32.947477 17824 leader_election.cc:302] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b2153ae384b54a2db238b23b788630ea, b3ba84eec8b7481b9dbf833fdcc24c96, ecea1053db6a46568968671e41dcb804; no voters:
I20260925 05:05:32.947793 17959 raft_consensus.cc:2781] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:32.948238 17959 raft_consensus.cc:686] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804 [term 1 LEADER]: Becoming Leader. State: Replica: ecea1053db6a46568968671e41dcb804, State: Running, Role: LEADER
I20260925 05:05:32.948823 17959 consensus_queue.cc:237] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ba84eec8b7481b9dbf833fdcc24c96" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 39763 } } peers { permanent_uuid: "ecea1053db6a46568968671e41dcb804" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35475 } } peers { permanent_uuid: "b2153ae384b54a2db238b23b788630ea" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 37291 } }
I20260925 05:05:32.953393 17477 catalog_manager.cc:5495] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecea1053db6a46568968671e41dcb804 (127.30.247.3). New cstate: current_term: 1 leader_uuid: "ecea1053db6a46568968671e41dcb804" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ba84eec8b7481b9dbf833fdcc24c96" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 39763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecea1053db6a46568968671e41dcb804" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 35475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2153ae384b54a2db238b23b788630ea" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 37291 } health_report { overall_health: UNKNOWN } } }
I20260925 05:05:32.973176 17949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:32.968076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33528 (local address 127.30.247.3:35475)
0925 05:05:32.968280 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:32.968284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:32.968422 (+ 138us) server_negotiation.cc:407] Connection header received
0925 05:05:32.968615 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:32.968619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:32.968689 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:32.968811 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:05:32.969130 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.969834 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.971119 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.971510 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.971600 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:32.972115 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:05:32.972713 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:05:32.972870 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:32.972951 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 05:05:32.972990 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 05:05:32.981326 17755 raft_consensus.cc:1261] T b241ecb8d98e4f7b80f0a1223bda5118 P b3ba84eec8b7481b9dbf833fdcc24c96 [term 1 FOLLOWER]: Refusing update from remote peer ecea1053db6a46568968671e41dcb804: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:32.981326 17621 raft_consensus.cc:1261] T b241ecb8d98e4f7b80f0a1223bda5118 P b2153ae384b54a2db238b23b788630ea [term 1 FOLLOWER]: Refusing update from remote peer ecea1053db6a46568968671e41dcb804: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:32.982474 17961 consensus_queue.cc:1035] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2153ae384b54a2db238b23b788630ea" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:32.982848 17959 consensus_queue.cc:1035] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3ba84eec8b7481b9dbf833fdcc24c96" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 39763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:32.995337 17966 mvcc.cc:205] Tried to move back new op lower bound from 7333120954280550400 to 7333120954161963008. Current Snapshot: MvccSnapshot[applied={T|T < 7333120954280550400}]
I20260925 05:05:33.007575 17476 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:33.007881 17476 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:33.011628 17476 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f21806a8d284ff699146a1f2068aa8b: Sending DeleteTablet for 3 replicas of tablet b241ecb8d98e4f7b80f0a1223bda5118
I20260925 05:05:33.012795 17735 tablet_service.cc:1514] Processing DeleteTablet for tablet b241ecb8d98e4f7b80f0a1223bda5118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:33 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:37962
I20260925 05:05:33.012912 17868 tablet_service.cc:1514] Processing DeleteTablet for tablet b241ecb8d98e4f7b80f0a1223bda5118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:33 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:33520
I20260925 05:05:33.012969 17601 tablet_service.cc:1514] Processing DeleteTablet for tablet b241ecb8d98e4f7b80f0a1223bda5118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:33 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:47360
I20260925 05:05:33.017026 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 17538
I20260925 05:05:33.027676 17974 tablet_replica.cc:330] T b241ecb8d98e4f7b80f0a1223bda5118 P b3ba84eec8b7481b9dbf833fdcc24c96: stopping tablet replica
I20260925 05:05:33.028244 17974 raft_consensus.cc:2229] T b241ecb8d98e4f7b80f0a1223bda5118 P b3ba84eec8b7481b9dbf833fdcc24c96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:33.028889 17974 raft_consensus.cc:2258] T b241ecb8d98e4f7b80f0a1223bda5118 P b3ba84eec8b7481b9dbf833fdcc24c96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:33.029567 17975 tablet_replica.cc:330] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804: stopping tablet replica
I20260925 05:05:33.030011 17975 raft_consensus.cc:2229] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:05:33.030699 17975 raft_consensus.cc:2258] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:33.034683 17974 ts_tablet_manager.cc:1905] T b241ecb8d98e4f7b80f0a1223bda5118 P b3ba84eec8b7481b9dbf833fdcc24c96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:33.035435 17975 ts_tablet_manager.cc:1905] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:33.036023 17976 tablet_replica.cc:330] T b241ecb8d98e4f7b80f0a1223bda5118 P b2153ae384b54a2db238b23b788630ea: stopping tablet replica
I20260925 05:05:33.041065 17976 raft_consensus.cc:2229] T b241ecb8d98e4f7b80f0a1223bda5118 P b2153ae384b54a2db238b23b788630ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:33.041170 17974 ts_tablet_manager.cc:1918] T b241ecb8d98e4f7b80f0a1223bda5118 P b3ba84eec8b7481b9dbf833fdcc24c96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:05:33.041353 17974 log.cc:1196] T b241ecb8d98e4f7b80f0a1223bda5118 P b3ba84eec8b7481b9dbf833fdcc24c96: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-1/wal/wals/b241ecb8d98e4f7b80f0a1223bda5118
I20260925 05:05:33.041800 17976 raft_consensus.cc:2258] T b241ecb8d98e4f7b80f0a1223bda5118 P b2153ae384b54a2db238b23b788630ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:33.041919 17974 ts_tablet_manager.cc:1939] T b241ecb8d98e4f7b80f0a1223bda5118 P b3ba84eec8b7481b9dbf833fdcc24c96: Deleting consensus metadata
I20260925 05:05:33.043581 17975 ts_tablet_manager.cc:1918] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:05:33.043800 17975 log.cc:1196] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-2/wal/wals/b241ecb8d98e4f7b80f0a1223bda5118
I20260925 05:05:33.043928 17463 catalog_manager.cc:4841] TS b3ba84eec8b7481b9dbf833fdcc24c96 (127.30.247.2:39763): tablet b241ecb8d98e4f7b80f0a1223bda5118 (table test-table [id=d494212e5dd74c97876936990755f2dc]) successfully deleted
I20260925 05:05:33.044574 17975 ts_tablet_manager.cc:1939] T b241ecb8d98e4f7b80f0a1223bda5118 P ecea1053db6a46568968671e41dcb804: Deleting consensus metadata
I20260925 05:05:33.046380 17461 catalog_manager.cc:4841] TS ecea1053db6a46568968671e41dcb804 (127.30.247.3:35475): tablet b241ecb8d98e4f7b80f0a1223bda5118 (table test-table [id=d494212e5dd74c97876936990755f2dc]) successfully deleted
I20260925 05:05:33.046602 17976 ts_tablet_manager.cc:1905] T b241ecb8d98e4f7b80f0a1223bda5118 P b2153ae384b54a2db238b23b788630ea: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 05:05:33 dist-test-slave-5n86 krb5kdc[17432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.1@KRBTEST.COM
I20260925 05:05:33.055924 17976 ts_tablet_manager.cc:1918] T b241ecb8d98e4f7b80f0a1223bda5118 P b2153ae384b54a2db238b23b788630ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:05:33.056152 17976 log.cc:1196] T b241ecb8d98e4f7b80f0a1223bda5118 P b2153ae384b54a2db238b23b788630ea: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312430481417-31708-0/minicluster-data/ts-0/wal/wals/b241ecb8d98e4f7b80f0a1223bda5118
I20260925 05:05:33.057078 17976 ts_tablet_manager.cc:1939] T b241ecb8d98e4f7b80f0a1223bda5118 P b2153ae384b54a2db238b23b788630ea: Deleting consensus metadata
I20260925 05:05:33.057431 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:33.017813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47374 (local address 127.30.247.1:37291)
0925 05:05:33.018036 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:33.018046 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:33.027296 (+ 9250us) server_negotiation.cc:407] Connection header received
0925 05:05:33.027494 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:33.027503 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:33.027612 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:33.027742 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:33.030419 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.032076 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.034699 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.035382 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.035502 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:33.049152 (+ 13650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:33.049175 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:33.049198 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:33.049258 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:33.053571 (+ 4313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:33.054535 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:33.054545 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:33.054549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:33.054690 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:33.055134 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:33.055140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:33.055145 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:33.055629 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:33.055935 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:33.056695 (+ 760us) server_negotiation.cc:299] Negotiation successful
0925 05:05:33.056950 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 05:05:33.059341 17661 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:33.059726 17464 catalog_manager.cc:4841] TS b2153ae384b54a2db238b23b788630ea (127.30.247.1:37291): tablet b241ecb8d98e4f7b80f0a1223bda5118 (table test-table [id=d494212e5dd74c97876936990755f2dc]) successfully deleted
I20260925 05:05:33.169694 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 17538
I20260925 05:05:33.195140 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 17672
Sep 25 05:05:33 dist-test-slave-5n86 krb5kdc[17432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.2@KRBTEST.COM
I20260925 05:05:33.213073 17946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:33.195964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37964 (local address 127.30.247.2:39763)
0925 05:05:33.196131 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:33.196135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:33.196187 (+ 52us) server_negotiation.cc:407] Connection header received
0925 05:05:33.196416 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:33.196420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:33.196482 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:33.196646 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:33.198425 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.199117 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.201315 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.201712 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.201816 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:33.207470 (+ 5654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:33.207483 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:33.207497 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:33.207528 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:33.210327 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:33.211147 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:33.211159 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:33.211161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:33.211283 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:33.211686 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:33.211689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:33.211690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:33.212050 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:33.212317 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:33.212651 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 05:05:33.212803 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:05:33.213693 17795 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:33.306401 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 17672
I20260925 05:05:33.330974 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 17805
Sep 25 05:05:33 dist-test-slave-5n86 krb5kdc[17432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.3@KRBTEST.COM
I20260925 05:05:33.349378 17949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:33.331731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33536 (local address 127.30.247.3:35475)
0925 05:05:33.331917 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:33.331922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:33.332046 (+ 124us) server_negotiation.cc:407] Connection header received
0925 05:05:33.332236 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:33.332240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:33.332308 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:33.332404 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:33.334222 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.335062 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.337088 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.337524 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.337618 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:33.343296 (+ 5678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:33.343311 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:33.343323 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:33.343355 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:33.346433 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:33.347099 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:33.347105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:33.347107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:33.347258 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:33.347641 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:33.347645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:33.347647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:33.348160 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:33.348464 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:33.348829 (+ 365us) server_negotiation.cc:299] Negotiation successful
0925 05:05:33.349049 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 05:05:33.350036 17928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:33.467608 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 17805
I20260925 05:05:33.486439 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 17445
I20260925 05:05:33.487465 17507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:33.597174 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 17445
2026-09-25T05:05:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4126 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T05:05:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:33Z Disabled control of system clock
I20260925 05:05:33.669087 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:34825
--webserver_interface=127.30.247.62
--webserver_port=0
--builtin_ntp_servers=127.30.247.20:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:34825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 05:05:33.988936 17985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:33.989210 17985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:33.989365 17985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:33.996441 17985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:33.996577 17985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:33.996640 17985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:33.996726 17985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:33.996773 17985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:34.005601 17985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:34825
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:34825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:34.006773 17985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:34.008464 17985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:34.019773 17990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:34.019690 17991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:34.019855 17993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:34.020872 17985 server_base.cc:1034] running on GCE node
I20260925 05:05:34.021608 17985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:34.022924 17985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:34.024121 17985 hybrid_clock.cc:648] HybridClock initialized: now 1790312734024088 us; error 30 us; skew 500 ppm
I20260925 05:05:34.024518 17985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:05:34.026798 17985 webserver.cc:460] Webserver started at http://127.30.247.62:41857/ using document root <none> and password file <none>
I20260925 05:05:34.027496 17985 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:34.027607 17985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:34.027931 17985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:34.030789 17985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "219bb1512be840ebb1273ea674bb97e5"
format_stamp: "Formatted at 2026-09-25 05:05:34 on dist-test-slave-5n86"
server_key: "8b6e791e32f807a6d8a7b03161b34a39"
server_key_iv: "0ca188e91d76d0293878f79162336aa0"
server_key_version: "encryptionkey@0"
I20260925 05:05:34.031582 17985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "219bb1512be840ebb1273ea674bb97e5"
format_stamp: "Formatted at 2026-09-25 05:05:34 on dist-test-slave-5n86"
server_key: "8b6e791e32f807a6d8a7b03161b34a39"
server_key_iv: "0ca188e91d76d0293878f79162336aa0"
server_key_version: "encryptionkey@0"
I20260925 05:05:34.036448 17985 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260925 05:05:34.039780 17999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:34.041026 17985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:05:34.041203 17985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "219bb1512be840ebb1273ea674bb97e5"
format_stamp: "Formatted at 2026-09-25 05:05:34 on dist-test-slave-5n86"
server_key: "8b6e791e32f807a6d8a7b03161b34a39"
server_key_iv: "0ca188e91d76d0293878f79162336aa0"
server_key_version: "encryptionkey@0"
I20260925 05:05:34.041419 17985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:34.052804 17985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:34.053557 17985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:34.053834 17985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:34.070972 17985 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:34825
I20260925 05:05:34.070992 18050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:34825 every 8 connection(s)
I20260925 05:05:34.072292 17985 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:34.076049 18051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:34.078039 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 17985
I20260925 05:05:34.078235 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:05:34.078706 31708 external_mini_cluster.cc:1428] Setting key a144533418d22d8cf28d9a1b4b996013
I20260925 05:05:34.087067 18051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 219bb1512be840ebb1273ea674bb97e5: Bootstrap starting.
I20260925 05:05:34.090093 18051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 219bb1512be840ebb1273ea674bb97e5: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:34.091210 18051 log.cc:824] T 00000000000000000000000000000000 P 219bb1512be840ebb1273ea674bb97e5: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:34.091735 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:34.080594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48242 (local address 127.30.247.62:34825)
0925 05:05:34.081581 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:34.081597 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:34.081640 (+ 43us) server_negotiation.cc:407] Connection header received
0925 05:05:34.083241 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:34.083267 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:34.083736 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:34.084139 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:34.084724 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:34.086002 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:34.087376 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:34.088019 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:34.088211 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:34.089433 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:34.089441 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:34.089455 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:34.089958 (+ 503us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:34.090047 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:34.090068 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:34.090194 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:34.090426 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 05:05:34.090714 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":340,"threads_started":1}
I20260925 05:05:34.094111 18051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 219bb1512be840ebb1273ea674bb97e5: No bootstrap required, opened a new log
I20260925 05:05:34.101361 18051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 219bb1512be840ebb1273ea674bb97e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219bb1512be840ebb1273ea674bb97e5" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 34825 } }
I20260925 05:05:34.101830 18051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 219bb1512be840ebb1273ea674bb97e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:34.101963 18051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 219bb1512be840ebb1273ea674bb97e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 219bb1512be840ebb1273ea674bb97e5, State: Initialized, Role: FOLLOWER
I20260925 05:05:34.102574 18051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 219bb1512be840ebb1273ea674bb97e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219bb1512be840ebb1273ea674bb97e5" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 34825 } }
I20260925 05:05:34.102777 18051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 219bb1512be840ebb1273ea674bb97e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:34.102914 18051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 219bb1512be840ebb1273ea674bb97e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:34.103107 18051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 219bb1512be840ebb1273ea674bb97e5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:34.105242 18051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 219bb1512be840ebb1273ea674bb97e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219bb1512be840ebb1273ea674bb97e5" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 34825 } }
I20260925 05:05:34.105719 18051 leader_election.cc:302] T 00000000000000000000000000000000 P 219bb1512be840ebb1273ea674bb97e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 219bb1512be840ebb1273ea674bb97e5; no voters:
I20260925 05:05:34.106257 18051 leader_election.cc:288] T 00000000000000000000000000000000 P 219bb1512be840ebb1273ea674bb97e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:34.106408 18056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 219bb1512be840ebb1273ea674bb97e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:34.107069 18056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 219bb1512be840ebb1273ea674bb97e5 [term 1 LEADER]: Becoming Leader. State: Replica: 219bb1512be840ebb1273ea674bb97e5, State: Running, Role: LEADER
I20260925 05:05:34.107537 18056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 219bb1512be840ebb1273ea674bb97e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219bb1512be840ebb1273ea674bb97e5" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 34825 } }
I20260925 05:05:34.108222 18051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 219bb1512be840ebb1273ea674bb97e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:34.110374 18058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 219bb1512be840ebb1273ea674bb97e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 219bb1512be840ebb1273ea674bb97e5. Latest consensus state: current_term: 1 leader_uuid: "219bb1512be840ebb1273ea674bb97e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219bb1512be840ebb1273ea674bb97e5" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 34825 } } }
I20260925 05:05:34.110790 18058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 219bb1512be840ebb1273ea674bb97e5 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:34.111272 18057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 219bb1512be840ebb1273ea674bb97e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "219bb1512be840ebb1273ea674bb97e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219bb1512be840ebb1273ea674bb97e5" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 34825 } } }
I20260925 05:05:34.111574 18057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 219bb1512be840ebb1273ea674bb97e5 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:34.112581 18064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:34.118888 18064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:34.127458 18064 catalog_manager.cc:1347] Generated new cluster ID: de397388c8f74139aef2495be57389e4
I20260925 05:05:34.127556 18064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:05:34.146854 18064 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:34.148089 18064 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:34.154712 18064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 219bb1512be840ebb1273ea674bb97e5: Generated new TSK 0
I20260925 05:05:34.155553 18064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:05:34.172431 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:0
--local_ip_for_outbound_sockets=127.30.247.1
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:34825
--builtin_ntp_servers=127.30.247.20:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 05:05:34.512846 18075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:34.513091 18075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:34.513222 18075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:34.520205 18075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:34.520295 18075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:34.520394 18075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:05:34.529349 18075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:34825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:34.530498 18075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:34.532200 18075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:34.545269 18080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:34.545226 18083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:34.545325 18081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:34.550022 18075 server_base.cc:1034] running on GCE node
I20260925 05:05:34.550654 18075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:34.551611 18075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:34.552820 18075 hybrid_clock.cc:648] HybridClock initialized: now 1790312734552734 us; error 75 us; skew 500 ppm
I20260925 05:05:34.553119 18075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:05:34.555223 18075 webserver.cc:460] Webserver started at http://127.30.247.1:39171/ using document root <none> and password file <none>
I20260925 05:05:34.555846 18075 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:34.555929 18075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:34.556180 18075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:34.559005 18075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-0/data/instance:
uuid: "08bc9c2f10c648328b42706984021c98"
format_stamp: "Formatted at 2026-09-25 05:05:34 on dist-test-slave-5n86"
server_key: "c4d8149b4abcceb97d1e4b83e7c2bc7f"
server_key_iv: "0b82605bb88629af98b6e2b4ae8ad23d"
server_key_version: "encryptionkey@0"
I20260925 05:05:34.559711 18075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance:
uuid: "08bc9c2f10c648328b42706984021c98"
format_stamp: "Formatted at 2026-09-25 05:05:34 on dist-test-slave-5n86"
server_key: "c4d8149b4abcceb97d1e4b83e7c2bc7f"
server_key_iv: "0b82605bb88629af98b6e2b4ae8ad23d"
server_key_version: "encryptionkey@0"
I20260925 05:05:34.564670 18075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 05:05:34.567966 18089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:34.569361 18075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:05:34.569538 18075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "08bc9c2f10c648328b42706984021c98"
format_stamp: "Formatted at 2026-09-25 05:05:34 on dist-test-slave-5n86"
server_key: "c4d8149b4abcceb97d1e4b83e7c2bc7f"
server_key_iv: "0b82605bb88629af98b6e2b4ae8ad23d"
server_key_version: "encryptionkey@0"
I20260925 05:05:34.569741 18075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:34.597540 18075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:34.598340 18075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:34.598639 18075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:34.599573 18075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:34.601009 18075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:34.601127 18075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:34.601262 18075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:34.601349 18075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:34.634639 18075 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:42625
I20260925 05:05:34.634728 18201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:42625 every 8 connection(s)
I20260925 05:05:34.635957 18075 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:05:34.639460 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 18075
I20260925 05:05:34.639668 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance
I20260925 05:05:34.640215 31708 external_mini_cluster.cc:1428] Setting key eef23eb16096e493573461a9cde89655
I20260925 05:05:34.642951 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:0
--local_ip_for_outbound_sockets=127.30.247.2
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:34825
--builtin_ntp_servers=127.30.247.20:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 05:05:34.649300 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:34.638542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:45719 (local address 127.30.247.62:34825)
0925 05:05:34.638996 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:34.639002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:34.640515 (+ 1513us) server_negotiation.cc:407] Connection header received
0925 05:05:34.642575 (+ 2060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:34.642580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:34.642670 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:34.642785 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:34.643902 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:34.644753 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:34.646506 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:34.646919 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:34.647052 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:34.648415 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:34.648420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:34.648423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:34.648441 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:34.648481 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:34.648493 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:34.648623 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:34.648843 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 05:05:34.648942 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":203,"threads_started":1}
I20260925 05:05:34.650324 18205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:34.639222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:34825 (local address 127.30.247.1:45719)
0925 05:05:34.640341 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:34.640402 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:34.642322 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:34.643041 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:34.643053 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:34.643189 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:34.643741 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:34.643757 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:34.644920 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:34.644924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:34.646361 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:34.646374 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:34.647136 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:34.647143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:34.647369 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:34.648089 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:05:34.648092 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:34.648131 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:05:34.648204 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:34.648621 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:34.648634 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:34.648644 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 05:05:34.648832 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 05:05:34.649228 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":372,"threads_started":1}
I20260925 05:05:34.651923 18202 heartbeater.cc:347] Connected to a master server at 127.30.247.62:34825
I20260925 05:05:34.652230 18202 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:34.653146 18202 heartbeater.cc:511] Master 127.30.247.62:34825 requested a full tablet report, sending...
I20260925 05:05:34.655315 18016 ts_manager.cc:194] Registered new tserver with Master: 08bc9c2f10c648328b42706984021c98 (127.30.247.1:42625)
I20260925 05:05:34.656930 18016 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.247.1:45719
W20260925 05:05:34.964020 18206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:34.964263 18206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:34.964399 18206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:34.971503 18206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:34.971590 18206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:34.971767 18206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:05:34.980726 18206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:34825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:34.981936 18206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:34.983661 18206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:34.996613 18211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:34.996685 18212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:34.996857 18214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:34.997641 18206 server_base.cc:1034] running on GCE node
I20260925 05:05:34.998214 18206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:34.999110 18206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:35.000331 18206 hybrid_clock.cc:648] HybridClock initialized: now 1790312735000290 us; error 53 us; skew 500 ppm
I20260925 05:05:35.000707 18206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:05:35.002837 18206 webserver.cc:460] Webserver started at http://127.30.247.2:45047/ using document root <none> and password file <none>
I20260925 05:05:35.003533 18206 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:35.003664 18206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:35.003983 18206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:35.007009 18206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-1/data/instance:
uuid: "d93ffb977c1a4d2aa1f9c1eb2d82416c"
format_stamp: "Formatted at 2026-09-25 05:05:34 on dist-test-slave-5n86"
server_key: "20a0900fc39dccbdff8fd7cafca80b29"
server_key_iv: "dba397f507f20384ee7429cef033cdc4"
server_key_version: "encryptionkey@0"
I20260925 05:05:35.007953 18206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance:
uuid: "d93ffb977c1a4d2aa1f9c1eb2d82416c"
format_stamp: "Formatted at 2026-09-25 05:05:34 on dist-test-slave-5n86"
server_key: "20a0900fc39dccbdff8fd7cafca80b29"
server_key_iv: "dba397f507f20384ee7429cef033cdc4"
server_key_version: "encryptionkey@0"
I20260925 05:05:35.013026 18206 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 05:05:35.016469 18220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:35.017802 18206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:05:35.017972 18206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "d93ffb977c1a4d2aa1f9c1eb2d82416c"
format_stamp: "Formatted at 2026-09-25 05:05:34 on dist-test-slave-5n86"
server_key: "20a0900fc39dccbdff8fd7cafca80b29"
server_key_iv: "dba397f507f20384ee7429cef033cdc4"
server_key_version: "encryptionkey@0"
I20260925 05:05:35.018177 18206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:35.035241 18206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:35.035967 18206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:35.036253 18206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:35.037225 18206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:35.038610 18206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:35.038695 18206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:35.038770 18206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:35.038823 18206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:35.071189 18206 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:40023
I20260925 05:05:35.071297 18332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:40023 every 8 connection(s)
I20260925 05:05:35.072525 18206 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:05:35.078266 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 18206
I20260925 05:05:35.078454 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance
I20260925 05:05:35.079017 31708 external_mini_cluster.cc:1428] Setting key 0a8aba25e9b7e697d5a5fde0d6822103
I20260925 05:05:35.082065 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:0
--local_ip_for_outbound_sockets=127.30.247.3
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:34825
--builtin_ntp_servers=127.30.247.20:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 05:05:35.087379 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.075288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:52687 (local address 127.30.247.62:34825)
0925 05:05:35.075458 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.075462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.077015 (+ 1553us) server_negotiation.cc:407] Connection header received
0925 05:05:35.079218 (+ 2203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.079224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.079306 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.079409 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:35.080620 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.081694 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.083969 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.084708 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.084842 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.086423 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:35.086432 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:35.086439 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:35.086463 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:35.086508 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:35.086527 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:35.086624 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.086881 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.087001 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 05:05:35.088433 18335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.075877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:34825 (local address 127.30.247.2:52687)
0925 05:05:35.076847 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:35.076891 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:35.078875 (+ 1984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:35.079667 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:35.079676 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:35.079760 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:35.080430 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.080447 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.081859 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.081865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.083807 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.083822 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.084939 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.084948 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.085250 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.086066 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:05:35.086070 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:35.086119 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:05:35.086214 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:35.086646 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:35.086658 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:35.086666 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 05:05:35.086841 (+ 175us) client_negotiation.cc:240] Negotiation successful
0925 05:05:35.087207 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":296,"threads_started":1}
I20260925 05:05:35.089931 18333 heartbeater.cc:347] Connected to a master server at 127.30.247.62:34825
I20260925 05:05:35.090272 18333 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:35.091121 18333 heartbeater.cc:511] Master 127.30.247.62:34825 requested a full tablet report, sending...
I20260925 05:05:35.092566 18016 ts_manager.cc:194] Registered new tserver with Master: d93ffb977c1a4d2aa1f9c1eb2d82416c (127.30.247.2:40023)
I20260925 05:05:35.093782 18016 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.247.2:52687
W20260925 05:05:35.402738 18336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:35.403013 18336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:35.403100 18336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:35.410264 18336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:35.410351 18336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:35.410457 18336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:05:35.419271 18336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:34825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:35.420434 18336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:35.422184 18336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:35.434552 18344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:35.434599 18342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:35.434623 18341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:35.435487 18336 server_base.cc:1034] running on GCE node
I20260925 05:05:35.436124 18336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:35.437100 18336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:35.438323 18336 hybrid_clock.cc:648] HybridClock initialized: now 1790312735438283 us; error 55 us; skew 500 ppm
I20260925 05:05:35.438634 18336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:05:35.440845 18336 webserver.cc:460] Webserver started at http://127.30.247.3:41465/ using document root <none> and password file <none>
I20260925 05:05:35.441553 18336 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:35.441685 18336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:35.441989 18336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:35.444823 18336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-2/data/instance:
uuid: "2b3d7ed86110452da378d332828a217b"
format_stamp: "Formatted at 2026-09-25 05:05:35 on dist-test-slave-5n86"
server_key: "c020bdeda7908aabd30e26d84e388875"
server_key_iv: "6a7d4fab7a3ffaa7eadc540e1041308d"
server_key_version: "encryptionkey@0"
I20260925 05:05:35.445616 18336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance:
uuid: "2b3d7ed86110452da378d332828a217b"
format_stamp: "Formatted at 2026-09-25 05:05:35 on dist-test-slave-5n86"
server_key: "c020bdeda7908aabd30e26d84e388875"
server_key_iv: "6a7d4fab7a3ffaa7eadc540e1041308d"
server_key_version: "encryptionkey@0"
I20260925 05:05:35.450588 18336 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:05:35.454126 18350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:35.455431 18336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:05:35.455613 18336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "2b3d7ed86110452da378d332828a217b"
format_stamp: "Formatted at 2026-09-25 05:05:35 on dist-test-slave-5n86"
server_key: "c020bdeda7908aabd30e26d84e388875"
server_key_iv: "6a7d4fab7a3ffaa7eadc540e1041308d"
server_key_version: "encryptionkey@0"
I20260925 05:05:35.455823 18336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:35.472374 18336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:35.473248 18336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:35.473541 18336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:35.474495 18336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:35.475867 18336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:35.475956 18336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:35.476105 18336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:35.476194 18336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:35.508955 18336 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:40419
I20260925 05:05:35.509083 18462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:40419 every 8 connection(s)
I20260925 05:05:35.510293 18336 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:05:35.517498 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 18336
I20260925 05:05:35.517680 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance
I20260925 05:05:35.518268 31708 external_mini_cluster.cc:1428] Setting key ea0a97c78dbaa081f9240cf26412a25f
I20260925 05:05:35.523932 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.512922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:59477 (local address 127.30.247.62:34825)
0925 05:05:35.513130 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.513134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.514539 (+ 1405us) server_negotiation.cc:407] Connection header received
0925 05:05:35.516400 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.516404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.516472 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.516634 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:35.517639 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.518781 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.521188 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.521652 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.521782 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.523137 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:35.523143 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:35.523147 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:35.523172 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:35.523202 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:35.523213 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:35.523302 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.523520 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.523611 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 05:05:35.524952 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.513467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:34825 (local address 127.30.247.3:59477)
0925 05:05:35.514377 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:35.514421 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:35.516133 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:35.516801 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:35.516809 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:35.516915 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:35.517468 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.517485 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.519014 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.519028 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.521043 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.521055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.521864 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.521873 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.522114 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.522833 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:05:35.522836 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:35.522875 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:05:35.522949 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:35.523301 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:35.523313 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:35.523323 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 05:05:35.523510 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 05:05:35.523873 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":293,"threads_started":1}
I20260925 05:05:35.526230 18463 heartbeater.cc:347] Connected to a master server at 127.30.247.62:34825
I20260925 05:05:35.526528 18463 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:35.527266 18463 heartbeater.cc:511] Master 127.30.247.62:34825 requested a full tablet report, sending...
I20260925 05:05:35.528687 18016 ts_manager.cc:194] Registered new tserver with Master: 2b3d7ed86110452da378d332828a217b (127.30.247.3:40419)
I20260925 05:05:35.529484 18016 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.247.3:59477
I20260925 05:05:35.533735 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:05:35.543761 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.538018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48250 (local address 127.30.247.62:34825)
0925 05:05:35.538308 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.538313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.538573 (+ 260us) server_negotiation.cc:407] Connection header received
0925 05:05:35.538710 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.538713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.538814 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.538907 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:35.539419 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.540085 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.541583 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.541970 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.542063 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.543133 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:35.543137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:35.543140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:35.543156 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:35.543195 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:35.543207 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:35.543326 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.543488 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.543537 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260925 05:05:35.547792 18016 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:05:35.554687 18016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:05:35.578325 18479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.568875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:42625 (local address 127.0.0.1:49694)
0925 05:05:35.569497 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:35.569518 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:35.570964 (+ 1446us) spinlock_profiling.cc:243] Waited 884 us on lock 0x61200002c1a8. stack: 00007f14ab3b366e 00007f14ab3b3430 00007f14aa43a5ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f14ab3a90fd 00007f14ab3a8d1e 00007f14be2b23cd 00007f14be2b236c 00007f14be27eb3b 00007f14b1db8536 00007f14b1db6d9e 00007f14b1e3edca 00007f14b1e3cf68 00007f14b1e7082f
0925 05:05:35.571005 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:35.571591 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:35.571595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:35.571617 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:35.571985 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.571993 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.573944 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.573952 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.575584 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.575603 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.577267 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.577272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.577468 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.577518 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 05:05:35.577638 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 05:05:35.577830 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":93,"spinlock_wait_cycles":2033408,"thread_start_us":314,"threads_started":1}
I20260925 05:05:35.578472 18475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.566835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:40419 (local address 127.0.0.1:32836)
0925 05:05:35.567723 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:35.567762 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:35.567960 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:35.570163 (+ 2203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:35.570172 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:35.570209 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:35.570619 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.570635 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.572744 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.572749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.575727 (+ 2978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.575742 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.577718 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.577724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.577893 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.577923 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 05:05:35.578021 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 05:05:35.578147 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":237,"thread_start_us":532,"threads_started":1}
I20260925 05:05:35.578819 18477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.567118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32836 (local address 127.30.247.3:40419)
0925 05:05:35.569232 (+ 2114us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.569241 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.569275 (+ 34us) server_negotiation.cc:407] Connection header received
0925 05:05:35.569373 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.569381 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.569635 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.569830 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:35.570879 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.572589 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.575968 (+ 3379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.577412 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.577521 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.577780 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.578080 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.578251 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1905,"thread_start_us":1743,"threads_started":1}
I20260925 05:05:35.579212 18480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.568940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49694 (local address 127.30.247.1:42625)
0925 05:05:35.570550 (+ 1610us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.570561 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.570591 (+ 30us) server_negotiation.cc:407] Connection header received
0925 05:05:35.571215 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.571223 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.571410 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.571593 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:35.572199 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.573609 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.575834 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.577133 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.577283 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.577532 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.578771 (+ 1239us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.578935 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1449,"thread_start_us":1338,"threads_started":1}
I20260925 05:05:35.579324 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.567692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56860 (local address 127.30.247.2:40023)
0925 05:05:35.568645 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.568654 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.568685 (+ 31us) server_negotiation.cc:407] Connection header received
0925 05:05:35.570091 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.570103 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.570359 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.570535 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:35.571265 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.572805 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.575968 (+ 3163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.577412 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.577535 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.577781 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.578753 (+ 972us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.578937 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":593,"threads_started":1}
I20260925 05:05:35.579737 18476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.567018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:40023 (local address 127.0.0.1:56860)
0925 05:05:35.568579 (+ 1561us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:35.568601 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:35.569845 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:35.570677 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:35.570682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:35.570712 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:35.571059 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.571067 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.572959 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.572966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.575727 (+ 2761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.575742 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.577910 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.577915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.578115 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.578146 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 05:05:35.578233 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 05:05:35.578375 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1406,"mutex_wait_us":155,"thread_start_us":1252,"threads_started":1}
I20260925 05:05:35.582858 18137 tablet_service.cc:1467] Processing CreateTablet for tablet 8749bc015f1a49ae8dcd51fc51c00fe8 (DEFAULT_TABLE table=test-table [id=623a8b5bcafe48fcbc93fd754939a0ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:35.582849 18398 tablet_service.cc:1467] Processing CreateTablet for tablet 8749bc015f1a49ae8dcd51fc51c00fe8 (DEFAULT_TABLE table=test-table [id=623a8b5bcafe48fcbc93fd754939a0ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:35.583441 18268 tablet_service.cc:1467] Processing CreateTablet for tablet 8749bc015f1a49ae8dcd51fc51c00fe8 (DEFAULT_TABLE table=test-table [id=623a8b5bcafe48fcbc93fd754939a0ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:35.584296 18137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8749bc015f1a49ae8dcd51fc51c00fe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:35.584295 18398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8749bc015f1a49ae8dcd51fc51c00fe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:35.584769 18268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8749bc015f1a49ae8dcd51fc51c00fe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:35.595948 18481 tablet_bootstrap.cc:492] T 8749bc015f1a49ae8dcd51fc51c00fe8 P d93ffb977c1a4d2aa1f9c1eb2d82416c: Bootstrap starting.
I20260925 05:05:35.599757 18481 tablet_bootstrap.cc:654] T 8749bc015f1a49ae8dcd51fc51c00fe8 P d93ffb977c1a4d2aa1f9c1eb2d82416c: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:35.600112 18482 tablet_bootstrap.cc:492] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98: Bootstrap starting.
I20260925 05:05:35.601140 18481 log.cc:824] T 8749bc015f1a49ae8dcd51fc51c00fe8 P d93ffb977c1a4d2aa1f9c1eb2d82416c: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:35.602139 18483 tablet_bootstrap.cc:492] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 2b3d7ed86110452da378d332828a217b: Bootstrap starting.
I20260925 05:05:35.603770 18481 tablet_bootstrap.cc:492] T 8749bc015f1a49ae8dcd51fc51c00fe8 P d93ffb977c1a4d2aa1f9c1eb2d82416c: No bootstrap required, opened a new log
I20260925 05:05:35.603791 18482 tablet_bootstrap.cc:654] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:35.604064 18481 ts_tablet_manager.cc:1397] T 8749bc015f1a49ae8dcd51fc51c00fe8 P d93ffb977c1a4d2aa1f9c1eb2d82416c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 05:05:35.605022 18482 log.cc:824] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:35.605556 18483 tablet_bootstrap.cc:654] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 2b3d7ed86110452da378d332828a217b: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:35.606583 18483 log.cc:824] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 2b3d7ed86110452da378d332828a217b: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:35.608399 18482 tablet_bootstrap.cc:492] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98: No bootstrap required, opened a new log
I20260925 05:05:35.608764 18482 ts_tablet_manager.cc:1397] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260925 05:05:35.612023 18481 raft_consensus.cc:348] T 8749bc015f1a49ae8dcd51fc51c00fe8 P d93ffb977c1a4d2aa1f9c1eb2d82416c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3d7ed86110452da378d332828a217b" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 40419 } } peers { permanent_uuid: "d93ffb977c1a4d2aa1f9c1eb2d82416c" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 40023 } } peers { permanent_uuid: "08bc9c2f10c648328b42706984021c98" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 42625 } }
I20260925 05:05:35.612382 18481 raft_consensus.cc:374] T 8749bc015f1a49ae8dcd51fc51c00fe8 P d93ffb977c1a4d2aa1f9c1eb2d82416c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:35.612478 18481 raft_consensus.cc:729] T 8749bc015f1a49ae8dcd51fc51c00fe8 P d93ffb977c1a4d2aa1f9c1eb2d82416c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d93ffb977c1a4d2aa1f9c1eb2d82416c, State: Initialized, Role: FOLLOWER
I20260925 05:05:35.613293 18481 consensus_queue.cc:260] T 8749bc015f1a49ae8dcd51fc51c00fe8 P d93ffb977c1a4d2aa1f9c1eb2d82416c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3d7ed86110452da378d332828a217b" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 40419 } } peers { permanent_uuid: "d93ffb977c1a4d2aa1f9c1eb2d82416c" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 40023 } } peers { permanent_uuid: "08bc9c2f10c648328b42706984021c98" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 42625 } }
I20260925 05:05:35.614461 18333 heartbeater.cc:503] Master 127.30.247.62:34825 was elected leader, sending a full tablet report...
I20260925 05:05:35.615316 18483 tablet_bootstrap.cc:492] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 2b3d7ed86110452da378d332828a217b: No bootstrap required, opened a new log
I20260925 05:05:35.615582 18483 ts_tablet_manager.cc:1397] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 2b3d7ed86110452da378d332828a217b: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260925 05:05:35.616407 18481 ts_tablet_manager.cc:1428] T 8749bc015f1a49ae8dcd51fc51c00fe8 P d93ffb977c1a4d2aa1f9c1eb2d82416c: Time spent starting tablet: real 0.012s user 0.005s sys 0.007s
I20260925 05:05:35.620066 18482 raft_consensus.cc:348] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3d7ed86110452da378d332828a217b" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 40419 } } peers { permanent_uuid: "d93ffb977c1a4d2aa1f9c1eb2d82416c" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 40023 } } peers { permanent_uuid: "08bc9c2f10c648328b42706984021c98" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 42625 } }
I20260925 05:05:35.620524 18482 raft_consensus.cc:374] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:35.620661 18482 raft_consensus.cc:729] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bc9c2f10c648328b42706984021c98, State: Initialized, Role: FOLLOWER
I20260925 05:05:35.621282 18482 consensus_queue.cc:260] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3d7ed86110452da378d332828a217b" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 40419 } } peers { permanent_uuid: "d93ffb977c1a4d2aa1f9c1eb2d82416c" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 40023 } } peers { permanent_uuid: "08bc9c2f10c648328b42706984021c98" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 42625 } }
I20260925 05:05:35.622422 18202 heartbeater.cc:503] Master 127.30.247.62:34825 was elected leader, sending a full tablet report...
I20260925 05:05:35.623085 18482 ts_tablet_manager.cc:1428] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260925 05:05:35.625373 18483 raft_consensus.cc:348] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 2b3d7ed86110452da378d332828a217b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3d7ed86110452da378d332828a217b" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 40419 } } peers { permanent_uuid: "d93ffb977c1a4d2aa1f9c1eb2d82416c" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 40023 } } peers { permanent_uuid: "08bc9c2f10c648328b42706984021c98" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 42625 } }
I20260925 05:05:35.625803 18483 raft_consensus.cc:374] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 2b3d7ed86110452da378d332828a217b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:35.625919 18483 raft_consensus.cc:729] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 2b3d7ed86110452da378d332828a217b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3d7ed86110452da378d332828a217b, State: Initialized, Role: FOLLOWER
I20260925 05:05:35.626607 18483 consensus_queue.cc:260] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 2b3d7ed86110452da378d332828a217b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3d7ed86110452da378d332828a217b" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 40419 } } peers { permanent_uuid: "d93ffb977c1a4d2aa1f9c1eb2d82416c" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 40023 } } peers { permanent_uuid: "08bc9c2f10c648328b42706984021c98" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 42625 } }
I20260925 05:05:35.627688 18463 heartbeater.cc:503] Master 127.30.247.62:34825 was elected leader, sending a full tablet report...
I20260925 05:05:35.628340 18483 ts_tablet_manager.cc:1428] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 2b3d7ed86110452da378d332828a217b: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260925 05:05:35.638222 18203 tablet.cc:2369] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:05:35.705979 18488 raft_consensus.cc:482] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:05:35.706336 18488 raft_consensus.cc:504] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3d7ed86110452da378d332828a217b" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 40419 } } peers { permanent_uuid: "d93ffb977c1a4d2aa1f9c1eb2d82416c" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 40023 } } peers { permanent_uuid: "08bc9c2f10c648328b42706984021c98" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 42625 } }
I20260925 05:05:35.708117 18488 leader_election.cc:288] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b3d7ed86110452da378d332828a217b (127.30.247.3:40419), d93ffb977c1a4d2aa1f9c1eb2d82416c (127.30.247.2:40023)
I20260925 05:05:35.715422 18477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.708665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:39483 (local address 127.30.247.3:40419)
0925 05:05:35.708886 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.708894 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.709868 (+ 974us) server_negotiation.cc:407] Connection header received
0925 05:05:35.710025 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.710031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.710109 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.710203 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:35.710823 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.711617 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.713705 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.714490 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.714617 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.714664 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.715039 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.715166 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:05:35.715451 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.708602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:55453 (local address 127.30.247.2:40023)
0925 05:05:35.708812 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.708821 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.710031 (+ 1210us) server_negotiation.cc:407] Connection header received
0925 05:05:35.710235 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.710241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.710335 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.710439 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:35.711064 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.711808 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.713773 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.714617 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.714782 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.714853 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.715101 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.715219 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 05:05:35.715525 18491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.708634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:40023 (local address 127.30.247.1:55453)
0925 05:05:35.709933 (+ 1299us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:35.709957 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:35.710085 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:35.710462 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:35.710465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:35.710484 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:35.710909 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.710923 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.711971 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.711977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.713567 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.713579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.714803 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.714807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.714920 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.714941 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 05:05:35.715049 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 05:05:35.715176 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1138,"mutex_wait_us":66,"thread_start_us":571,"threads_started":1}
I20260925 05:05:35.715575 18490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.708632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:40419 (local address 127.30.247.1:39483)
0925 05:05:35.709752 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:35.709768 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:35.709897 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:35.710231 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:35.710236 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:35.710262 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:35.710670 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.710681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.711761 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.711768 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.713504 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.713516 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.714677 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.714686 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.714861 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.714908 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 05:05:35.714981 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 05:05:35.715134 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"mutex_wait_us":6,"thread_start_us":611,"threads_started":1}
I20260925 05:05:35.716640 18418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8749bc015f1a49ae8dcd51fc51c00fe8" candidate_uuid: "08bc9c2f10c648328b42706984021c98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3d7ed86110452da378d332828a217b" is_pre_election: true
I20260925 05:05:35.716719 18288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8749bc015f1a49ae8dcd51fc51c00fe8" candidate_uuid: "08bc9c2f10c648328b42706984021c98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d93ffb977c1a4d2aa1f9c1eb2d82416c" is_pre_election: true
I20260925 05:05:35.717192 18418 raft_consensus.cc:2454] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 2b3d7ed86110452da378d332828a217b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08bc9c2f10c648328b42706984021c98 in term 0.
I20260925 05:05:35.717228 18288 raft_consensus.cc:2454] T 8749bc015f1a49ae8dcd51fc51c00fe8 P d93ffb977c1a4d2aa1f9c1eb2d82416c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08bc9c2f10c648328b42706984021c98 in term 0.
I20260925 05:05:35.717895 18091 leader_election.cc:302] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bc9c2f10c648328b42706984021c98, 2b3d7ed86110452da378d332828a217b; no voters:
I20260925 05:05:35.718248 18488 raft_consensus.cc:2781] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:05:35.718405 18488 raft_consensus.cc:482] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:05:35.718513 18488 raft_consensus.cc:3037] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:35.720697 18488 raft_consensus.cc:504] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3d7ed86110452da378d332828a217b" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 40419 } } peers { permanent_uuid: "d93ffb977c1a4d2aa1f9c1eb2d82416c" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 40023 } } peers { permanent_uuid: "08bc9c2f10c648328b42706984021c98" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 42625 } }
I20260925 05:05:35.721271 18488 leader_election.cc:288] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98 [CANDIDATE]: Term 1 election: Requested vote from peers 2b3d7ed86110452da378d332828a217b (127.30.247.3:40419), d93ffb977c1a4d2aa1f9c1eb2d82416c (127.30.247.2:40023)
I20260925 05:05:35.721755 18288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8749bc015f1a49ae8dcd51fc51c00fe8" candidate_uuid: "08bc9c2f10c648328b42706984021c98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d93ffb977c1a4d2aa1f9c1eb2d82416c"
I20260925 05:05:35.721782 18418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8749bc015f1a49ae8dcd51fc51c00fe8" candidate_uuid: "08bc9c2f10c648328b42706984021c98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3d7ed86110452da378d332828a217b"
I20260925 05:05:35.721966 18288 raft_consensus.cc:3037] T 8749bc015f1a49ae8dcd51fc51c00fe8 P d93ffb977c1a4d2aa1f9c1eb2d82416c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:35.722029 18418 raft_consensus.cc:3037] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 2b3d7ed86110452da378d332828a217b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:35.724493 18288 raft_consensus.cc:2454] T 8749bc015f1a49ae8dcd51fc51c00fe8 P d93ffb977c1a4d2aa1f9c1eb2d82416c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08bc9c2f10c648328b42706984021c98 in term 1.
I20260925 05:05:35.724493 18418 raft_consensus.cc:2454] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 2b3d7ed86110452da378d332828a217b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08bc9c2f10c648328b42706984021c98 in term 1.
I20260925 05:05:35.724960 18092 leader_election.cc:302] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bc9c2f10c648328b42706984021c98, d93ffb977c1a4d2aa1f9c1eb2d82416c; no voters:
I20260925 05:05:35.725255 18488 raft_consensus.cc:2781] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:35.725730 18488 raft_consensus.cc:686] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98 [term 1 LEADER]: Becoming Leader. State: Replica: 08bc9c2f10c648328b42706984021c98, State: Running, Role: LEADER
I20260925 05:05:35.726192 18488 consensus_queue.cc:237] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3d7ed86110452da378d332828a217b" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 40419 } } peers { permanent_uuid: "d93ffb977c1a4d2aa1f9c1eb2d82416c" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 40023 } } peers { permanent_uuid: "08bc9c2f10c648328b42706984021c98" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 42625 } }
I20260925 05:05:35.730831 18015 catalog_manager.cc:5495] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08bc9c2f10c648328b42706984021c98 (127.30.247.1). New cstate: current_term: 1 leader_uuid: "08bc9c2f10c648328b42706984021c98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3d7ed86110452da378d332828a217b" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 40419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d93ffb977c1a4d2aa1f9c1eb2d82416c" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08bc9c2f10c648328b42706984021c98" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 42625 } health_report { overall_health: HEALTHY } } }
W20260925 05:05:35.761960 18464 tablet.cc:2369] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 2b3d7ed86110452da378d332828a217b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:05:35.773213 18480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.767469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49710 (local address 127.30.247.1:42625)
0925 05:05:35.767682 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.767688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.767788 (+ 100us) server_negotiation.cc:407] Connection header received
0925 05:05:35.768044 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.768048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.768136 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.768283 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:05:35.768703 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.769405 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.770775 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.771178 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.771309 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.771920 (+ 611us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:05:35.772414 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:05:35.772812 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.772912 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.772981 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 05:05:35.781065 18418 raft_consensus.cc:1261] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 2b3d7ed86110452da378d332828a217b [term 1 FOLLOWER]: Refusing update from remote peer 08bc9c2f10c648328b42706984021c98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:35.781062 18288 raft_consensus.cc:1261] T 8749bc015f1a49ae8dcd51fc51c00fe8 P d93ffb977c1a4d2aa1f9c1eb2d82416c [term 1 FOLLOWER]: Refusing update from remote peer 08bc9c2f10c648328b42706984021c98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:35.781991 18492 consensus_queue.cc:1035] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3d7ed86110452da378d332828a217b" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 40419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:35.782450 18488 consensus_queue.cc:1035] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d93ffb977c1a4d2aa1f9c1eb2d82416c" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:35.806131 18015 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:35.806439 18015 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:35.810307 18015 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 219bb1512be840ebb1273ea674bb97e5: Sending DeleteTablet for 3 replicas of tablet 8749bc015f1a49ae8dcd51fc51c00fe8
I20260925 05:05:35.811512 18398 tablet_service.cc:1514] Processing DeleteTablet for tablet 8749bc015f1a49ae8dcd51fc51c00fe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:35 UTC) from {username='slave'} at 127.0.0.1:32836
I20260925 05:05:35.811712 18268 tablet_service.cc:1514] Processing DeleteTablet for tablet 8749bc015f1a49ae8dcd51fc51c00fe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:35 UTC) from {username='slave'} at 127.0.0.1:56860
I20260925 05:05:35.812669 18506 tablet_replica.cc:330] T 8749bc015f1a49ae8dcd51fc51c00fe8 P d93ffb977c1a4d2aa1f9c1eb2d82416c: stopping tablet replica
I20260925 05:05:35.812997 18137 tablet_service.cc:1514] Processing DeleteTablet for tablet 8749bc015f1a49ae8dcd51fc51c00fe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:35 UTC) from {username='slave'} at 127.0.0.1:49694
I20260925 05:05:35.813266 18506 raft_consensus.cc:2229] T 8749bc015f1a49ae8dcd51fc51c00fe8 P d93ffb977c1a4d2aa1f9c1eb2d82416c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:35.813468 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 18075
I20260925 05:05:35.813865 18506 raft_consensus.cc:2258] T 8749bc015f1a49ae8dcd51fc51c00fe8 P d93ffb977c1a4d2aa1f9c1eb2d82416c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:35.822531 18506 ts_tablet_manager.cc:1905] T 8749bc015f1a49ae8dcd51fc51c00fe8 P d93ffb977c1a4d2aa1f9c1eb2d82416c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:35.827764 18505 tablet_replica.cc:330] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 2b3d7ed86110452da378d332828a217b: stopping tablet replica
I20260925 05:05:35.828208 18505 raft_consensus.cc:2229] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 2b3d7ed86110452da378d332828a217b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:35.828716 18505 raft_consensus.cc:2258] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 2b3d7ed86110452da378d332828a217b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:35.831319 18505 ts_tablet_manager.cc:1905] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 2b3d7ed86110452da378d332828a217b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:35.832549 18480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.814862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49712 (local address 127.30.247.1:42625)
0925 05:05:35.815036 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.815043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.815059 (+ 16us) server_negotiation.cc:407] Connection header received
0925 05:05:35.815107 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.815118 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.815184 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.815290 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:35.816313 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.817574 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.828134 (+ 10560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.828864 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.828970 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.830623 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:35.830634 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:35.830651 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:35.831571 (+ 920us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:35.831646 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:35.831679 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:35.831842 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.832034 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.832136 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 05:05:35.834164 18197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:35.838083 18506 ts_tablet_manager.cc:1918] T 8749bc015f1a49ae8dcd51fc51c00fe8 P d93ffb977c1a4d2aa1f9c1eb2d82416c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:05:35.838292 18506 log.cc:1196] T 8749bc015f1a49ae8dcd51fc51c00fe8 P d93ffb977c1a4d2aa1f9c1eb2d82416c: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-1/wal/wals/8749bc015f1a49ae8dcd51fc51c00fe8
I20260925 05:05:35.838949 18506 ts_tablet_manager.cc:1939] T 8749bc015f1a49ae8dcd51fc51c00fe8 P d93ffb977c1a4d2aa1f9c1eb2d82416c: Deleting consensus metadata
I20260925 05:05:35.839732 18505 ts_tablet_manager.cc:1918] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 2b3d7ed86110452da378d332828a217b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:05:35.839901 18505 log.cc:1196] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 2b3d7ed86110452da378d332828a217b: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312430481417-31708-0/minicluster-data/ts-2/wal/wals/8749bc015f1a49ae8dcd51fc51c00fe8
I20260925 05:05:35.840507 18505 ts_tablet_manager.cc:1939] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 2b3d7ed86110452da378d332828a217b: Deleting consensus metadata
I20260925 05:05:35.841897 18002 catalog_manager.cc:4841] TS d93ffb977c1a4d2aa1f9c1eb2d82416c (127.30.247.2:40023): tablet 8749bc015f1a49ae8dcd51fc51c00fe8 (table test-table [id=623a8b5bcafe48fcbc93fd754939a0ea]) successfully deleted
I20260925 05:05:35.842896 18001 catalog_manager.cc:4841] TS 2b3d7ed86110452da378d332828a217b (127.30.247.3:40419): tablet 8749bc015f1a49ae8dcd51fc51c00fe8 (table test-table [id=623a8b5bcafe48fcbc93fd754939a0ea]) successfully deleted
I20260925 05:05:35.935535 18507 tablet_replica.cc:330] T 8749bc015f1a49ae8dcd51fc51c00fe8 P 08bc9c2f10c648328b42706984021c98: stopping tablet replica
I20260925 05:05:35.935801 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 18075
I20260925 05:05:35.953663 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 18206
W20260925 05:05:35.953608 18002 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.247.1:42625 (error 108)
I20260925 05:05:35.955519 18476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.955052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:42625 (local address 127.0.0.1:49720)
0925 05:05:35.955228 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:35.955346 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.1:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 05:05:35.955734 18002 catalog_manager.cc:4584] TS 08bc9c2f10c648328b42706984021c98 (127.30.247.1:42625): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8749bc015f1a49ae8dcd51fc51c00fe8: Network error: Client connection negotiation failed: client connection to 127.30.247.1:42625: connect: Connection refused (error 111)
I20260925 05:05:35.962955 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.954453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56862 (local address 127.30.247.2:40023)
0925 05:05:35.954649 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.954658 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.954818 (+ 160us) server_negotiation.cc:407] Connection header received
0925 05:05:35.955046 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.955050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.955182 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.955292 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:35.955722 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.956522 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.958601 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.959352 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.959505 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.960954 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:35.960963 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:35.960976 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:35.961752 (+ 776us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:35.961861 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:35.961888 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:35.961997 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.962471 (+ 474us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.962589 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 05:05:35.963577 18328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:36.009316 18476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:36.008872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:42625 (local address 127.0.0.1:49734)
0925 05:05:36.009072 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:36.009159 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.1:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 05:05:36.010401 18476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:36.010069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:42625 (local address 127.0.0.1:49736)
0925 05:05:36.010215 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:36.010285 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.1:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 05:05:36.052435 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 18206
I20260925 05:05:36.069751 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 18336
I20260925 05:05:36.071722 18476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:36.071239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:42625 (local address 127.0.0.1:49752)
0925 05:05:36.071465 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:36.071547 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.1:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260925 05:05:36.072796 18476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:36.072399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:42625 (local address 127.0.0.1:49762)
0925 05:05:36.072586 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:36.072665 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.1:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 05:05:36.077585 18477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:36.070574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32842 (local address 127.30.247.3:40419)
0925 05:05:36.070799 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:36.070806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:36.070880 (+ 74us) server_negotiation.cc:407] Connection header received
0925 05:05:36.071086 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:36.071092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:36.071188 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:36.071299 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:36.071690 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:36.072769 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:36.074593 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:36.074974 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:36.075067 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:36.076145 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:36.076153 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:36.076168 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:36.076876 (+ 708us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:36.076922 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:36.076942 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:36.077028 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:36.077175 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 05:05:36.077263 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 05:05:36.078219 18458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:36.155081 18476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:36.154550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:42625 (local address 127.0.0.1:49766)
0925 05:05:36.154774 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:36.154897 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.1:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 05:05:36.156173 18476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:36.155754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:42625 (local address 127.0.0.1:49778)
0925 05:05:36.155968 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:36.156046 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.1:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260925 05:05:36.168128 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 18336
I20260925 05:05:36.184667 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 17985
I20260925 05:05:36.185796 18046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:36.293911 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 17985
2026-09-25T05:05:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2695 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:05:36 dist-test-slave-5n86 krb5kdc[18516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:05:36 dist-test-slave-5n86 krb5kdc[18516](info): set up 2 sockets
Sep 25 05:05:36 dist-test-slave-5n86 krb5kdc[18516](info): commencing operation
krb5kdc: starting...
W20260925 05:05:37.389070 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:05:37 dist-test-slave-5n86 krb5kdc[18516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:05:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:05:37.633054 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:40133
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:40133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:37.954300 18529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:37.954569 18529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:37.954734 18529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:37.964363 18529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:37.964527 18529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:37.964660 18529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:37.964754 18529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:37.964852 18529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:37.980283 18529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:40133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:40133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:37.982157 18529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:37.983775 18529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:37.995507 18537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:37.995548 18535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:37.995728 18534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:37.996712 18529 server_base.cc:1034] running on GCE node
I20260925 05:05:37.997447 18529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:37.998826 18529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:38.000061 18529 hybrid_clock.cc:648] HybridClock initialized: now 1790312737999989 us; error 62 us; skew 500 ppm
I20260925 05:05:38.000471 18529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:05:37 dist-test-slave-5n86 krb5kdc[18516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312737, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:38.003734 18529 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:05:38.005241 18529 webserver.cc:460] Webserver started at http://127.30.247.62:41805/ using document root <none> and password file <none>
I20260925 05:05:38.005890 18529 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:38.005993 18529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:38.006237 18529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:38.009174 18529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "864fd8259e794f7db3f954fcef76d588"
format_stamp: "Formatted at 2026-09-25 05:05:38 on dist-test-slave-5n86"
server_key: "afd25b6c2eb139a46558937c59b2b75f"
server_key_iv: "243e3db8e18350636e6345d2093294dd"
server_key_version: "encryptionkey@0"
I20260925 05:05:38.009852 18529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "864fd8259e794f7db3f954fcef76d588"
format_stamp: "Formatted at 2026-09-25 05:05:38 on dist-test-slave-5n86"
server_key: "afd25b6c2eb139a46558937c59b2b75f"
server_key_iv: "243e3db8e18350636e6345d2093294dd"
server_key_version: "encryptionkey@0"
I20260925 05:05:38.014992 18529 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 05:05:38.018383 18544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:38.019708 18529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:05:38.019896 18529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "864fd8259e794f7db3f954fcef76d588"
format_stamp: "Formatted at 2026-09-25 05:05:38 on dist-test-slave-5n86"
server_key: "afd25b6c2eb139a46558937c59b2b75f"
server_key_iv: "243e3db8e18350636e6345d2093294dd"
server_key_version: "encryptionkey@0"
I20260925 05:05:38.020095 18529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:38.036368 18529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:38.040331 18529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:38.040670 18529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:38.059100 18529 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:40133
I20260925 05:05:38.059098 18595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:40133 every 8 connection(s)
I20260925 05:05:38.060403 18529 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:38.064144 18596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:38.067063 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 18529
I20260925 05:05:38.067253 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:05:38.067782 31708 external_mini_cluster.cc:1428] Setting key 85f87146049b138e4f72b95673989d75
I20260925 05:05:38.076206 18596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 864fd8259e794f7db3f954fcef76d588: Bootstrap starting.
I20260925 05:05:38.079943 18596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 864fd8259e794f7db3f954fcef76d588: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:38.081285 18596 log.cc:824] T 00000000000000000000000000000000 P 864fd8259e794f7db3f954fcef76d588: Log is configured to *not* fsync() on all Append() calls
Sep 25 05:05:38 dist-test-slave-5n86 krb5kdc[18516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:05:38.084649 18596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 864fd8259e794f7db3f954fcef76d588: No bootstrap required, opened a new log
I20260925 05:05:38.091249 18599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:38.069619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52824 (local address 127.30.247.62:40133)
0925 05:05:38.070374 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:38.070384 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:38.070423 (+ 39us) server_negotiation.cc:407] Connection header received
0925 05:05:38.072063 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:38.072082 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:38.072443 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:38.072835 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:38.074601 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.075837 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.077563 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.078227 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.078357 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:38.083566 (+ 5209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:38.083574 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:38.083587 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:38.083630 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:38.086945 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:38.087837 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:38.087844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:38.087853 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:38.088030 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:38.088516 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:38.088519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:38.088522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:38.089128 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:38.089355 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:38.089799 (+ 444us) server_negotiation.cc:299] Negotiation successful
0925 05:05:38.090176 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":377,"threads_started":1}
I20260925 05:05:38.093971 18596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 864fd8259e794f7db3f954fcef76d588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864fd8259e794f7db3f954fcef76d588" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40133 } }
I20260925 05:05:38.094429 18596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 864fd8259e794f7db3f954fcef76d588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:38.094568 18596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 864fd8259e794f7db3f954fcef76d588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 864fd8259e794f7db3f954fcef76d588, State: Initialized, Role: FOLLOWER
I20260925 05:05:38.095155 18596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 864fd8259e794f7db3f954fcef76d588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864fd8259e794f7db3f954fcef76d588" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40133 } }
I20260925 05:05:38.095408 18596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 864fd8259e794f7db3f954fcef76d588 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:38.095546 18596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 864fd8259e794f7db3f954fcef76d588 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:38.095726 18596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 864fd8259e794f7db3f954fcef76d588 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:38.098182 18596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 864fd8259e794f7db3f954fcef76d588 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864fd8259e794f7db3f954fcef76d588" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40133 } }
I20260925 05:05:38.098670 18596 leader_election.cc:302] T 00000000000000000000000000000000 P 864fd8259e794f7db3f954fcef76d588 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 864fd8259e794f7db3f954fcef76d588; no voters:
I20260925 05:05:38.099263 18596 leader_election.cc:288] T 00000000000000000000000000000000 P 864fd8259e794f7db3f954fcef76d588 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:38.099400 18601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 864fd8259e794f7db3f954fcef76d588 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:38.100172 18601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 864fd8259e794f7db3f954fcef76d588 [term 1 LEADER]: Becoming Leader. State: Replica: 864fd8259e794f7db3f954fcef76d588, State: Running, Role: LEADER
I20260925 05:05:38.100778 18601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 864fd8259e794f7db3f954fcef76d588 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864fd8259e794f7db3f954fcef76d588" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40133 } }
I20260925 05:05:38.101362 18596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 864fd8259e794f7db3f954fcef76d588 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:38.103526 18603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 864fd8259e794f7db3f954fcef76d588 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 864fd8259e794f7db3f954fcef76d588. Latest consensus state: current_term: 1 leader_uuid: "864fd8259e794f7db3f954fcef76d588" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864fd8259e794f7db3f954fcef76d588" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40133 } } }
I20260925 05:05:38.103950 18603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 864fd8259e794f7db3f954fcef76d588 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:38.103783 18602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 864fd8259e794f7db3f954fcef76d588 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "864fd8259e794f7db3f954fcef76d588" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864fd8259e794f7db3f954fcef76d588" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40133 } } }
I20260925 05:05:38.104180 18602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 864fd8259e794f7db3f954fcef76d588 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:38.107025 18609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:38.111639 18609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:38.119925 18609 catalog_manager.cc:1347] Generated new cluster ID: 8519d35555c64120b71eba078ae9d86d
I20260925 05:05:38.120030 18609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:05:38.132382 18609 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:38.134246 18609 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:38.144873 18609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 864fd8259e794f7db3f954fcef76d588: Generated new TSK 0
I20260925 05:05:38.145777 18609 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
Entry for principal kudu/127.30.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab.
I20260925 05:05:38.208464 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:0
--local_ip_for_outbound_sockets=127.30.247.1
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:40133
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:38.533030 18622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:38.533319 18622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:38.533478 18622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:38.540875 18622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:38.541002 18622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:38.541184 18622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:05:38.551311 18622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.1.keytab
--principal=kudu/127.30.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:40133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:38.552728 18622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:38.554047 18622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:38.567200 18627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:38.567252 18628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:38.567288 18630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:38.568233 18622 server_base.cc:1034] running on GCE node
I20260925 05:05:38.568915 18622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:38.569880 18622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:38.571102 18622 hybrid_clock.cc:648] HybridClock initialized: now 1790312738571113 us; error 102 us; skew 500 ppm
I20260925 05:05:38.571404 18622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:05:38 dist-test-slave-5n86 krb5kdc[18516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312738, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:38.574555 18622 init.cc:377] Logged in from keytab as kudu/127.30.247.1@KRBTEST.COM (short username kudu)
I20260925 05:05:38.575949 18622 webserver.cc:460] Webserver started at http://127.30.247.1:42517/ using document root <none> and password file <none>
I20260925 05:05:38.576642 18622 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:38.576745 18622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:38.577008 18622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:38.579756 18622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-0/data/instance:
uuid: "4289df0eaeff4aee87f9e47842052527"
format_stamp: "Formatted at 2026-09-25 05:05:38 on dist-test-slave-5n86"
server_key: "f6ae2b3d3e61d4e7cf1574d8192acb74"
server_key_iv: "089eaff52df9945925d15f699d13433e"
server_key_version: "encryptionkey@0"
I20260925 05:05:38.580439 18622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance:
uuid: "4289df0eaeff4aee87f9e47842052527"
format_stamp: "Formatted at 2026-09-25 05:05:38 on dist-test-slave-5n86"
server_key: "f6ae2b3d3e61d4e7cf1574d8192acb74"
server_key_iv: "089eaff52df9945925d15f699d13433e"
server_key_version: "encryptionkey@0"
I20260925 05:05:38.585547 18622 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 05:05:38.588901 18637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:38.590250 18622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:05:38.590456 18622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "4289df0eaeff4aee87f9e47842052527"
format_stamp: "Formatted at 2026-09-25 05:05:38 on dist-test-slave-5n86"
server_key: "f6ae2b3d3e61d4e7cf1574d8192acb74"
server_key_iv: "089eaff52df9945925d15f699d13433e"
server_key_version: "encryptionkey@0"
I20260925 05:05:38.590672 18622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:38.632769 18622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:38.636935 18622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:38.637248 18622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:38.638211 18622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:38.639600 18622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:38.639700 18622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:38.639848 18622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:38.639918 18622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:38.673560 18622 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:46289
I20260925 05:05:38.673645 18749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:46289 every 8 connection(s)
I20260925 05:05:38.674860 18622 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:05:38.682569 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 18622
I20260925 05:05:38.682740 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance
I20260925 05:05:38.683241 31708 external_mini_cluster.cc:1428] Setting key dc840117144bfecde53f5ef23300e15e
Sep 25 05:05:38 dist-test-slave-5n86 krb5kdc[18516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312738, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.1@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:05:38.695299 18752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:38.677480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:52241 (local address 127.30.247.62:40133)
0925 05:05:38.677949 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:38.677954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:38.678865 (+ 911us) server_negotiation.cc:407] Connection header received
0925 05:05:38.680657 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:38.680661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:38.680725 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:38.680821 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:38.682223 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.683041 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.685010 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.685495 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.685601 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:38.689623 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:38.689628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:38.689632 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:38.689661 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:38.692173 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:38.692991 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:38.692996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:38.692998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:38.693115 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:38.693528 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:38.693533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:38.693536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:38.693794 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:38.693949 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:38.694688 (+ 739us) server_negotiation.cc:299] Negotiation successful
0925 05:05:38.694915 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":219,"threads_started":1}
I20260925 05:05:38.696206 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:38.678067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40133 (local address 127.30.247.1:52241)
0925 05:05:38.678711 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:38.678748 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:38.680370 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:38.681050 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:38.681058 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:38.681541 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:38.682077 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:38.682092 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.683186 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.683191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:38.684819 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:38.684838 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.685722 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.685730 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:38.686051 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:38.686751 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:38.686785 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:38.689418 (+ 2633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:38.692324 (+ 2906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:38.692332 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:38.692349 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:38.692856 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:38.693225 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:38.693230 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:38.693233 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:38.693411 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:38.693929 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:38.693937 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:38.694322 (+ 385us) client_negotiation.cc:769] Sending connection context
0925 05:05:38.694572 (+ 250us) client_negotiation.cc:240] Negotiation successful
0925 05:05:38.695083 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":261,"threads_started":1}
I20260925 05:05:38.697705 18750 heartbeater.cc:347] Connected to a master server at 127.30.247.62:40133
I20260925 05:05:38.698028 18750 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:38.698853 18750 heartbeater.cc:511] Master 127.30.247.62:40133 requested a full tablet report, sending...
I20260925 05:05:38.701001 18561 ts_manager.cc:194] Registered new tserver with Master: 4289df0eaeff4aee87f9e47842052527 (127.30.247.1:46289)
I20260925 05:05:38.702548 18561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.1@KRBTEST.COM'} at 127.30.247.1:52241
WARNING: no policy specified for kudu/127.30.247.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
Entry for principal kudu/127.30.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab.
I20260925 05:05:38.739778 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:0
--local_ip_for_outbound_sockets=127.30.247.2
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:40133
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:39.059379 18756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:39.059641 18756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:39.059791 18756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:39.066967 18756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:39.067063 18756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:39.067232 18756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:05:39.076881 18756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.2.keytab
--principal=kudu/127.30.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:40133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:39.078063 18756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:39.079327 18756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:39.092576 18761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:39.092621 18762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:39.092698 18764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:39.093617 18756 server_base.cc:1034] running on GCE node
I20260925 05:05:39.094210 18756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:39.095130 18756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:39.096374 18756 hybrid_clock.cc:648] HybridClock initialized: now 1790312739096331 us; error 27 us; skew 500 ppm
I20260925 05:05:39.096738 18756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:05:39 dist-test-slave-5n86 krb5kdc[18516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312739, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:39.099799 18756 init.cc:377] Logged in from keytab as kudu/127.30.247.2@KRBTEST.COM (short username kudu)
I20260925 05:05:39.101272 18756 webserver.cc:460] Webserver started at http://127.30.247.2:40259/ using document root <none> and password file <none>
I20260925 05:05:39.101886 18756 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:39.101989 18756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:39.102233 18756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:39.105043 18756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-1/data/instance:
uuid: "c590c18e610f46709ffb0b0139774e2f"
format_stamp: "Formatted at 2026-09-25 05:05:39 on dist-test-slave-5n86"
server_key: "dd57a21ab0809aab766a74f125872bd6"
server_key_iv: "3db0158e1c53b48415ff1967eb7bb733"
server_key_version: "encryptionkey@0"
I20260925 05:05:39.105800 18756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance:
uuid: "c590c18e610f46709ffb0b0139774e2f"
format_stamp: "Formatted at 2026-09-25 05:05:39 on dist-test-slave-5n86"
server_key: "dd57a21ab0809aab766a74f125872bd6"
server_key_iv: "3db0158e1c53b48415ff1967eb7bb733"
server_key_version: "encryptionkey@0"
I20260925 05:05:39.110805 18756 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:05:39.114317 18771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:39.115626 18756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:05:39.115752 18756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "c590c18e610f46709ffb0b0139774e2f"
format_stamp: "Formatted at 2026-09-25 05:05:39 on dist-test-slave-5n86"
server_key: "dd57a21ab0809aab766a74f125872bd6"
server_key_iv: "3db0158e1c53b48415ff1967eb7bb733"
server_key_version: "encryptionkey@0"
I20260925 05:05:39.115902 18756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:39.142208 18756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:39.146481 18756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:39.146713 18756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:39.147600 18756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:39.148903 18756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:39.148986 18756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:39.149068 18756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:39.149114 18756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:39.182632 18756 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:35877
I20260925 05:05:39.182729 18883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:35877 every 8 connection(s)
I20260925 05:05:39.183903 18756 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:05:39.192786 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 18756
I20260925 05:05:39.192978 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance
I20260925 05:05:39.193411 31708 external_mini_cluster.cc:1428] Setting key f77d88309aaab0815c405edb0fad01fc
Sep 25 05:05:39 dist-test-slave-5n86 krb5kdc[18516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312739, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.2@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:05:39.204981 18752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:39.186453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:47161 (local address 127.30.247.62:40133)
0925 05:05:39.186644 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:39.186649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:39.187742 (+ 1093us) server_negotiation.cc:407] Connection header received
0925 05:05:39.189628 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:39.189632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:39.189704 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:39.189795 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:39.191353 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.192082 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.194048 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.194503 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.194692 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:39.198199 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:39.198207 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:39.198215 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:39.198258 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:39.201911 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:39.202739 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:39.202747 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:39.202753 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:39.202913 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:39.203313 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:39.203320 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:39.203326 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:39.203718 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:39.203871 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:39.204561 (+ 690us) server_negotiation.cc:299] Negotiation successful
0925 05:05:39.204747 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 05:05:39.206382 18886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:39.186837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40133 (local address 127.30.247.2:47161)
0925 05:05:39.187561 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:39.187598 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:39.189277 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:39.190133 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:39.190141 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:39.190652 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:39.191185 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:39.191202 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.192220 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.192224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:39.193855 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:39.193871 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.194746 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.194752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:39.195019 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:39.195800 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:39.195822 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:39.197968 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:39.202076 (+ 4108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:39.202097 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:39.202117 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:39.202613 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:39.203026 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:39.203029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:39.203031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:39.203199 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:39.203835 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:39.203840 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:39.204151 (+ 311us) client_negotiation.cc:769] Sending connection context
0925 05:05:39.204362 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 05:05:39.204916 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":304,"threads_started":1}
I20260925 05:05:39.207954 18884 heartbeater.cc:347] Connected to a master server at 127.30.247.62:40133
I20260925 05:05:39.208266 18884 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:39.209008 18884 heartbeater.cc:511] Master 127.30.247.62:40133 requested a full tablet report, sending...
I20260925 05:05:39.210450 18561 ts_manager.cc:194] Registered new tserver with Master: c590c18e610f46709ffb0b0139774e2f (127.30.247.2:35877)
I20260925 05:05:39.211648 18561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.2@KRBTEST.COM'} at 127.30.247.2:47161
WARNING: no policy specified for kudu/127.30.247.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
Entry for principal kudu/127.30.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab.
I20260925 05:05:39.250108 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:0
--local_ip_for_outbound_sockets=127.30.247.3
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:40133
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:39.562589 18889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:39.562896 18889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:39.563063 18889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:39.570091 18889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:39.570235 18889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:39.570407 18889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:05:39.579668 18889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.3.keytab
--principal=kudu/127.30.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:40133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:39.580924 18889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:39.582223 18889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:39.595110 18895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:39.595077 18894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:39.595156 18897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:39.596369 18889 server_base.cc:1034] running on GCE node
I20260925 05:05:39.596943 18889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:39.597834 18889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:39.599048 18889 hybrid_clock.cc:648] HybridClock initialized: now 1790312739598993 us; error 53 us; skew 500 ppm
I20260925 05:05:39.599336 18889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:05:39 dist-test-slave-5n86 krb5kdc[18516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312739, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:39.602442 18889 init.cc:377] Logged in from keytab as kudu/127.30.247.3@KRBTEST.COM (short username kudu)
I20260925 05:05:39.603834 18889 webserver.cc:460] Webserver started at http://127.30.247.3:40863/ using document root <none> and password file <none>
I20260925 05:05:39.604455 18889 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:39.604610 18889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:39.604909 18889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:39.607789 18889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-2/data/instance:
uuid: "3433635518fb4a66af65fc2b3599f0f1"
format_stamp: "Formatted at 2026-09-25 05:05:39 on dist-test-slave-5n86"
server_key: "5e34cefb9ca27ef4a3473ad334946353"
server_key_iv: "07ad74852aa0b33537ea21171d575e5b"
server_key_version: "encryptionkey@0"
I20260925 05:05:39.608505 18889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance:
uuid: "3433635518fb4a66af65fc2b3599f0f1"
format_stamp: "Formatted at 2026-09-25 05:05:39 on dist-test-slave-5n86"
server_key: "5e34cefb9ca27ef4a3473ad334946353"
server_key_iv: "07ad74852aa0b33537ea21171d575e5b"
server_key_version: "encryptionkey@0"
I20260925 05:05:39.613816 18889 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 05:05:39.617216 18904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:39.618583 18889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 05:05:39.618760 18889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "3433635518fb4a66af65fc2b3599f0f1"
format_stamp: "Formatted at 2026-09-25 05:05:39 on dist-test-slave-5n86"
server_key: "5e34cefb9ca27ef4a3473ad334946353"
server_key_iv: "07ad74852aa0b33537ea21171d575e5b"
server_key_version: "encryptionkey@0"
I20260925 05:05:39.618956 18889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:39.655421 18889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:39.659744 18889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:39.660037 18889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:39.661036 18889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:39.662457 18889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:39.662566 18889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:39.662707 18889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:39.662777 18889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:39.695351 18889 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:41855
I20260925 05:05:39.695403 19016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:41855 every 8 connection(s)
I20260925 05:05:39.696666 18889 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:05:39.702651 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 18889
I20260925 05:05:39.702858 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance
I20260925 05:05:39.703298 31708 external_mini_cluster.cc:1428] Setting key 741ee4d1b68854de896d10f91ebe4979
I20260925 05:05:39.706065 18750 heartbeater.cc:503] Master 127.30.247.62:40133 was elected leader, sending a full tablet report...
Sep 25 05:05:39 dist-test-slave-5n86 krb5kdc[18516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312739, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.3@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:05:39.715794 18752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:39.699141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:46815 (local address 127.30.247.62:40133)
0925 05:05:39.699342 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:39.699347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:39.700365 (+ 1018us) server_negotiation.cc:407] Connection header received
0925 05:05:39.702346 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:39.702352 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:39.702467 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:39.702581 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:39.704073 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.704902 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.706729 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.707322 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.707497 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:39.710642 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:39.710647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:39.710650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:39.710682 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:39.713263 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:39.713947 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:39.713950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:39.713952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:39.714067 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:39.714427 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:39.714430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:39.714432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:39.714682 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:39.714790 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:39.715410 (+ 620us) server_negotiation.cc:299] Negotiation successful
0925 05:05:39.715563 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 05:05:39.716784 19019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:39.699557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:40133 (local address 127.30.247.3:46815)
0925 05:05:39.700203 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:39.700242 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:39.702065 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:39.702812 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:39.702826 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:39.703351 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:39.703903 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:39.703926 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.705054 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.705058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:39.706566 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:39.706586 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.707577 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.707585 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:39.707791 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:39.708373 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:39.708394 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:39.710464 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:39.713399 (+ 2935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:39.713405 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:39.713416 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:39.713816 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:39.714156 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:39.714159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:39.714161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:39.714329 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:39.714782 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:39.714787 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:39.715085 (+ 298us) client_negotiation.cc:769] Sending connection context
0925 05:05:39.715312 (+ 227us) client_negotiation.cc:240] Negotiation successful
0925 05:05:39.715653 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":262,"threads_started":1}
I20260925 05:05:39.718254 19017 heartbeater.cc:347] Connected to a master server at 127.30.247.62:40133
I20260925 05:05:39.718516 19017 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:39.719236 19017 heartbeater.cc:511] Master 127.30.247.62:40133 requested a full tablet report, sending...
I20260925 05:05:39.720692 18561 ts_manager.cc:194] Registered new tserver with Master: 3433635518fb4a66af65fc2b3599f0f1 (127.30.247.3:41855)
I20260925 05:05:39.721539 18561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.247.3@KRBTEST.COM'} at 127.30.247.3:46815
I20260925 05:05:39.730149 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:05:39.746842 18752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:39.734315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49594 (local address 127.30.247.62:40133)
0925 05:05:39.734563 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:39.734570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:39.734877 (+ 307us) server_negotiation.cc:407] Connection header received
0925 05:05:39.735057 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:39.735062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:39.735126 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:39.735227 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:39.736692 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.737407 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.738705 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.739142 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.739275 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:39.741606 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:39.741611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:39.741615 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:39.741644 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:39.744136 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:39.744829 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:39.744835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:39.744839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:39.744985 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:39.745396 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:39.745401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:39.745404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:39.745709 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:39.745984 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:39.746328 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 05:05:39.746546 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 05:05:39.750630 18561 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:05:39.757977 18561 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:05:39.783242 19029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:39.769812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:46289 (local address 127.0.0.1:50576)
0925 05:05:39.771399 (+ 1587us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:39.771440 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:39.771686 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:39.772627 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:39.772633 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:39.772662 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:39.773125 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:39.773149 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.775899 (+ 2750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.775907 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:39.779105 (+ 3198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:39.779127 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.782162 (+ 3035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.782171 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:39.782397 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:39.782445 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 05:05:39.782568 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 05:05:39.782770 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1360,"thread_start_us":1210,"threads_started":1}
I20260925 05:05:39.783519 19031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:39.770095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50576 (local address 127.30.247.1:46289)
0925 05:05:39.771066 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:39.771079 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:39.771540 (+ 461us) server_negotiation.cc:407] Connection header received
0925 05:05:39.771997 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:39.772006 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:39.772265 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:39.773344 (+ 1079us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:39.773653 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.775695 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.779365 (+ 3670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.781416 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.781559 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:39.781860 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:39.782857 (+ 997us) server_negotiation.cc:299] Negotiation successful
0925 05:05:39.783084 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":554,"threads_started":1}
I20260925 05:05:39.785985 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:39.770451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44276 (local address 127.30.247.2:35877)
0925 05:05:39.771445 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:39.771457 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:39.772208 (+ 751us) server_negotiation.cc:407] Connection header received
0925 05:05:39.772311 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:39.772319 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:39.773942 (+ 1623us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:39.774157 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:39.775250 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.778551 (+ 3301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.783394 (+ 4843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.784900 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.785049 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:39.785250 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:39.785515 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 05:05:39.785685 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":841,"thread_start_us":759,"threads_started":1}
I20260925 05:05:39.786777 19030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:39.769909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:35877 (local address 127.0.0.1:44276)
0925 05:05:39.772033 (+ 2124us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:39.772050 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:39.772147 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:39.774428 (+ 2281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:39.774435 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:39.774470 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:39.775003 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:39.775017 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.778751 (+ 3734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.778759 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:39.783200 (+ 4441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:39.783218 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.785125 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.785132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:39.785316 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:39.785356 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 05:05:39.785454 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 05:05:39.785602 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2013,"thread_start_us":1872,"threads_started":1}
I20260925 05:05:39.787918 18685 tablet_service.cc:1467] Processing CreateTablet for tablet 6e00b9c539444f47a724c658f4f1cffc (DEFAULT_TABLE table=test-table [id=ba6dab424fa6429fb00b5b43c2cc8248]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:39.789538 18685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e00b9c539444f47a724c658f4f1cffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:39.790452 18819 tablet_service.cc:1467] Processing CreateTablet for tablet 6e00b9c539444f47a724c658f4f1cffc (DEFAULT_TABLE table=test-table [id=ba6dab424fa6429fb00b5b43c2cc8248]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:39.791107 19033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:39.772282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:41855 (local address 127.0.0.1:48942)
0925 05:05:39.783750 (+ 11468us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:39.783767 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:39.783892 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:39.784615 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:39.784619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:39.784643 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:39.784935 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:39.784944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.786796 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.786801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:39.788446 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:39.788458 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.790330 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.790337 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:39.790522 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:39.790553 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 05:05:39.790667 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 05:05:39.790785 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11333,"thread_start_us":11243,"threads_started":1}
I20260925 05:05:39.791230 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:39.772703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48942 (local address 127.30.247.3:41855)
0925 05:05:39.782698 (+ 9995us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:39.782710 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:39.783871 (+ 1161us) server_negotiation.cc:407] Connection header received
0925 05:05:39.784157 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:39.784169 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:39.784407 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:39.784623 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:39.785160 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.786655 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.789197 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.790130 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.790270 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:39.790486 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:39.790707 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 05:05:39.790857 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9746,"thread_start_us":9551,"threads_started":1}
I20260925 05:05:39.791756 18819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e00b9c539444f47a724c658f4f1cffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:39.795749 18952 tablet_service.cc:1467] Processing CreateTablet for tablet 6e00b9c539444f47a724c658f4f1cffc (DEFAULT_TABLE table=test-table [id=ba6dab424fa6429fb00b5b43c2cc8248]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:39.796900 18952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e00b9c539444f47a724c658f4f1cffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:39.804258 19035 tablet_bootstrap.cc:492] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527: Bootstrap starting.
I20260925 05:05:39.806301 19036 tablet_bootstrap.cc:492] T 6e00b9c539444f47a724c658f4f1cffc P c590c18e610f46709ffb0b0139774e2f: Bootstrap starting.
I20260925 05:05:39.808368 19035 tablet_bootstrap.cc:654] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:39.809012 19037 tablet_bootstrap.cc:492] T 6e00b9c539444f47a724c658f4f1cffc P 3433635518fb4a66af65fc2b3599f0f1: Bootstrap starting.
I20260925 05:05:39.809661 19036 tablet_bootstrap.cc:654] T 6e00b9c539444f47a724c658f4f1cffc P c590c18e610f46709ffb0b0139774e2f: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:39.809784 19035 log.cc:824] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:39.810865 19036 log.cc:824] T 6e00b9c539444f47a724c658f4f1cffc P c590c18e610f46709ffb0b0139774e2f: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:39.812216 19037 tablet_bootstrap.cc:654] T 6e00b9c539444f47a724c658f4f1cffc P 3433635518fb4a66af65fc2b3599f0f1: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:39.812716 19035 tablet_bootstrap.cc:492] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527: No bootstrap required, opened a new log
I20260925 05:05:39.812968 19035 ts_tablet_manager.cc:1397] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260925 05:05:39.813345 19037 log.cc:824] T 6e00b9c539444f47a724c658f4f1cffc P 3433635518fb4a66af65fc2b3599f0f1: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:39.813952 19036 tablet_bootstrap.cc:492] T 6e00b9c539444f47a724c658f4f1cffc P c590c18e610f46709ffb0b0139774e2f: No bootstrap required, opened a new log
I20260925 05:05:39.814210 19036 ts_tablet_manager.cc:1397] T 6e00b9c539444f47a724c658f4f1cffc P c590c18e610f46709ffb0b0139774e2f: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 05:05:39.816133 19037 tablet_bootstrap.cc:492] T 6e00b9c539444f47a724c658f4f1cffc P 3433635518fb4a66af65fc2b3599f0f1: No bootstrap required, opened a new log
I20260925 05:05:39.816376 19037 ts_tablet_manager.cc:1397] T 6e00b9c539444f47a724c658f4f1cffc P 3433635518fb4a66af65fc2b3599f0f1: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260925 05:05:39.821157 19035 raft_consensus.cc:348] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4289df0eaeff4aee87f9e47842052527" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46289 } } peers { permanent_uuid: "c590c18e610f46709ffb0b0139774e2f" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35877 } } peers { permanent_uuid: "3433635518fb4a66af65fc2b3599f0f1" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41855 } }
I20260925 05:05:39.821516 19035 raft_consensus.cc:374] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:39.821614 19035 raft_consensus.cc:729] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4289df0eaeff4aee87f9e47842052527, State: Initialized, Role: FOLLOWER
I20260925 05:05:39.822225 19035 consensus_queue.cc:260] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4289df0eaeff4aee87f9e47842052527" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46289 } } peers { permanent_uuid: "c590c18e610f46709ffb0b0139774e2f" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35877 } } peers { permanent_uuid: "3433635518fb4a66af65fc2b3599f0f1" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41855 } }
I20260925 05:05:39.824766 19036 raft_consensus.cc:348] T 6e00b9c539444f47a724c658f4f1cffc P c590c18e610f46709ffb0b0139774e2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4289df0eaeff4aee87f9e47842052527" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46289 } } peers { permanent_uuid: "c590c18e610f46709ffb0b0139774e2f" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35877 } } peers { permanent_uuid: "3433635518fb4a66af65fc2b3599f0f1" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41855 } }
I20260925 05:05:39.825881 19035 ts_tablet_manager.cc:1428] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
I20260925 05:05:39.826591 19036 raft_consensus.cc:374] T 6e00b9c539444f47a724c658f4f1cffc P c590c18e610f46709ffb0b0139774e2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:39.826762 19036 raft_consensus.cc:729] T 6e00b9c539444f47a724c658f4f1cffc P c590c18e610f46709ffb0b0139774e2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c590c18e610f46709ffb0b0139774e2f, State: Initialized, Role: FOLLOWER
I20260925 05:05:39.826659 19037 raft_consensus.cc:348] T 6e00b9c539444f47a724c658f4f1cffc P 3433635518fb4a66af65fc2b3599f0f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4289df0eaeff4aee87f9e47842052527" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46289 } } peers { permanent_uuid: "c590c18e610f46709ffb0b0139774e2f" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35877 } } peers { permanent_uuid: "3433635518fb4a66af65fc2b3599f0f1" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41855 } }
I20260925 05:05:39.827055 19037 raft_consensus.cc:374] T 6e00b9c539444f47a724c658f4f1cffc P 3433635518fb4a66af65fc2b3599f0f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:39.827160 19037 raft_consensus.cc:729] T 6e00b9c539444f47a724c658f4f1cffc P 3433635518fb4a66af65fc2b3599f0f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3433635518fb4a66af65fc2b3599f0f1, State: Initialized, Role: FOLLOWER
I20260925 05:05:39.827494 19036 consensus_queue.cc:260] T 6e00b9c539444f47a724c658f4f1cffc P c590c18e610f46709ffb0b0139774e2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4289df0eaeff4aee87f9e47842052527" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46289 } } peers { permanent_uuid: "c590c18e610f46709ffb0b0139774e2f" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35877 } } peers { permanent_uuid: "3433635518fb4a66af65fc2b3599f0f1" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41855 } }
I20260925 05:05:39.827693 19037 consensus_queue.cc:260] T 6e00b9c539444f47a724c658f4f1cffc P 3433635518fb4a66af65fc2b3599f0f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4289df0eaeff4aee87f9e47842052527" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46289 } } peers { permanent_uuid: "c590c18e610f46709ffb0b0139774e2f" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35877 } } peers { permanent_uuid: "3433635518fb4a66af65fc2b3599f0f1" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41855 } }
I20260925 05:05:39.829450 19017 heartbeater.cc:503] Master 127.30.247.62:40133 was elected leader, sending a full tablet report...
I20260925 05:05:39.829578 19037 ts_tablet_manager.cc:1428] T 6e00b9c539444f47a724c658f4f1cffc P 3433635518fb4a66af65fc2b3599f0f1: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260925 05:05:39.835104 18884 heartbeater.cc:503] Master 127.30.247.62:40133 was elected leader, sending a full tablet report...
I20260925 05:05:39.836043 19036 ts_tablet_manager.cc:1428] T 6e00b9c539444f47a724c658f4f1cffc P c590c18e610f46709ffb0b0139774e2f: Time spent starting tablet: real 0.022s user 0.015s sys 0.004s
I20260925 05:05:39.845500 19041 raft_consensus.cc:482] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:05:39.845777 19041 raft_consensus.cc:504] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4289df0eaeff4aee87f9e47842052527" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46289 } } peers { permanent_uuid: "c590c18e610f46709ffb0b0139774e2f" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35877 } } peers { permanent_uuid: "3433635518fb4a66af65fc2b3599f0f1" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41855 } }
I20260925 05:05:39.847399 19041 leader_election.cc:288] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c590c18e610f46709ffb0b0139774e2f (127.30.247.2:35877), 3433635518fb4a66af65fc2b3599f0f1 (127.30.247.3:41855)
I20260925 05:05:39.854666 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:39.847824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:35045 (local address 127.30.247.2:35877)
0925 05:05:39.848040 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:39.848046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:39.848239 (+ 193us) server_negotiation.cc:407] Connection header received
0925 05:05:39.848501 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:39.848510 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:39.848630 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:39.848751 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:39.849396 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.850357 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.852728 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.853813 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.853978 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:39.854051 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:39.854316 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 05:05:39.854447 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 05:05:39.854772 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:39.847563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:35877 (local address 127.30.247.1:35045)
0925 05:05:39.848140 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:39.848156 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:39.848317 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:39.848788 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:39.848795 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:39.848831 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:39.849251 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:39.849262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.850485 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.850489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:39.852573 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:39.852589 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.854031 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.854036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:39.854157 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:39.854181 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 05:05:39.854262 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 05:05:39.854387 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":316,"threads_started":1}
I20260925 05:05:39.855301 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:39.848514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:41255 (local address 127.30.247.3:41855)
0925 05:05:39.848694 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:39.848698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:39.849214 (+ 516us) server_negotiation.cc:407] Connection header received
0925 05:05:39.849419 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:39.849424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:39.849502 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:39.849606 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:39.850060 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.850910 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.853101 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.854275 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.854440 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:39.854503 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:39.854888 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 05:05:39.855037 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 05:05:39.855789 19045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:39.848299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:41855 (local address 127.30.247.1:41255)
0925 05:05:39.849082 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:39.849099 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:39.849238 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:39.849624 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:39.849627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:39.849646 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:39.849923 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:39.849930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.851042 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.851051 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:39.852963 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:39.852973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.854491 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.854499 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:39.854680 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:39.854713 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 05:05:39.855471 (+ 758us) client_negotiation.cc:240] Negotiation successful
0925 05:05:39.855579 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":450,"threads_started":1}
I20260925 05:05:39.855820 18839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e00b9c539444f47a724c658f4f1cffc" candidate_uuid: "4289df0eaeff4aee87f9e47842052527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c590c18e610f46709ffb0b0139774e2f" is_pre_election: true
I20260925 05:05:39.856400 18839 raft_consensus.cc:2454] T 6e00b9c539444f47a724c658f4f1cffc P c590c18e610f46709ffb0b0139774e2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4289df0eaeff4aee87f9e47842052527 in term 0.
I20260925 05:05:39.856819 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e00b9c539444f47a724c658f4f1cffc" candidate_uuid: "4289df0eaeff4aee87f9e47842052527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3433635518fb4a66af65fc2b3599f0f1" is_pre_election: true
I20260925 05:05:39.857146 18639 leader_election.cc:302] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4289df0eaeff4aee87f9e47842052527, c590c18e610f46709ffb0b0139774e2f; no voters:
I20260925 05:05:39.857262 18972 raft_consensus.cc:2454] T 6e00b9c539444f47a724c658f4f1cffc P 3433635518fb4a66af65fc2b3599f0f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4289df0eaeff4aee87f9e47842052527 in term 0.
I20260925 05:05:39.857542 19041 raft_consensus.cc:2781] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:05:39.857689 19041 raft_consensus.cc:482] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:05:39.857827 19041 raft_consensus.cc:3037] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:39.859969 19041 raft_consensus.cc:504] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4289df0eaeff4aee87f9e47842052527" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46289 } } peers { permanent_uuid: "c590c18e610f46709ffb0b0139774e2f" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35877 } } peers { permanent_uuid: "3433635518fb4a66af65fc2b3599f0f1" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41855 } }
I20260925 05:05:39.860522 19041 leader_election.cc:288] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527 [CANDIDATE]: Term 1 election: Requested vote from peers c590c18e610f46709ffb0b0139774e2f (127.30.247.2:35877), 3433635518fb4a66af65fc2b3599f0f1 (127.30.247.3:41855)
I20260925 05:05:39.860999 18839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e00b9c539444f47a724c658f4f1cffc" candidate_uuid: "4289df0eaeff4aee87f9e47842052527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c590c18e610f46709ffb0b0139774e2f"
I20260925 05:05:39.861056 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e00b9c539444f47a724c658f4f1cffc" candidate_uuid: "4289df0eaeff4aee87f9e47842052527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3433635518fb4a66af65fc2b3599f0f1"
I20260925 05:05:39.861194 18839 raft_consensus.cc:3037] T 6e00b9c539444f47a724c658f4f1cffc P c590c18e610f46709ffb0b0139774e2f [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:39.861275 18972 raft_consensus.cc:3037] T 6e00b9c539444f47a724c658f4f1cffc P 3433635518fb4a66af65fc2b3599f0f1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:39.863547 18839 raft_consensus.cc:2454] T 6e00b9c539444f47a724c658f4f1cffc P c590c18e610f46709ffb0b0139774e2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4289df0eaeff4aee87f9e47842052527 in term 1.
I20260925 05:05:39.863626 18972 raft_consensus.cc:2454] T 6e00b9c539444f47a724c658f4f1cffc P 3433635518fb4a66af65fc2b3599f0f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4289df0eaeff4aee87f9e47842052527 in term 1.
I20260925 05:05:39.863973 18639 leader_election.cc:302] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4289df0eaeff4aee87f9e47842052527, c590c18e610f46709ffb0b0139774e2f; no voters:
I20260925 05:05:39.864269 19041 raft_consensus.cc:2781] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:39.864750 19041 raft_consensus.cc:686] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527 [term 1 LEADER]: Becoming Leader. State: Replica: 4289df0eaeff4aee87f9e47842052527, State: Running, Role: LEADER
I20260925 05:05:39.865263 19041 consensus_queue.cc:237] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4289df0eaeff4aee87f9e47842052527" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46289 } } peers { permanent_uuid: "c590c18e610f46709ffb0b0139774e2f" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35877 } } peers { permanent_uuid: "3433635518fb4a66af65fc2b3599f0f1" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41855 } }
I20260925 05:05:39.869777 18561 catalog_manager.cc:5495] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4289df0eaeff4aee87f9e47842052527 (127.30.247.1). New cstate: current_term: 1 leader_uuid: "4289df0eaeff4aee87f9e47842052527" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4289df0eaeff4aee87f9e47842052527" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 46289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c590c18e610f46709ffb0b0139774e2f" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3433635518fb4a66af65fc2b3599f0f1" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41855 } health_report { overall_health: UNKNOWN } } }
I20260925 05:05:39.902534 19031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:39.897125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50592 (local address 127.30.247.1:46289)
0925 05:05:39.897352 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:39.897357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:39.897375 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:05:39.897545 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:39.897550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:39.897624 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:39.897744 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:05:39.898140 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.898857 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.900232 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.900755 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.900857 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:39.901356 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:05:39.901872 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:05:39.901984 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:39.902140 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 05:05:39.902233 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 05:05:39.910776 18972 raft_consensus.cc:1261] T 6e00b9c539444f47a724c658f4f1cffc P 3433635518fb4a66af65fc2b3599f0f1 [term 1 FOLLOWER]: Refusing update from remote peer 4289df0eaeff4aee87f9e47842052527: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:39.910820 18839 raft_consensus.cc:1261] T 6e00b9c539444f47a724c658f4f1cffc P c590c18e610f46709ffb0b0139774e2f [term 1 FOLLOWER]: Refusing update from remote peer 4289df0eaeff4aee87f9e47842052527: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:39.911692 19046 consensus_queue.cc:1035] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c590c18e610f46709ffb0b0139774e2f" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 35877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:39.912091 19041 consensus_queue.cc:1035] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3433635518fb4a66af65fc2b3599f0f1" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 41855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:39.938789 18561 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:39.939091 18561 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:39.942982 18561 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 864fd8259e794f7db3f954fcef76d588: Sending DeleteTablet for 3 replicas of tablet 6e00b9c539444f47a724c658f4f1cffc
I20260925 05:05:39.944033 18685 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e00b9c539444f47a724c658f4f1cffc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:39 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:50576
I20260925 05:05:39.944132 18952 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e00b9c539444f47a724c658f4f1cffc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:39 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:48942
I20260925 05:05:39.944875 18819 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e00b9c539444f47a724c658f4f1cffc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:39 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:44276
I20260925 05:05:39.945020 18951 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e00b9c539444f47a724c658f4f1cffc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:39 UTC) from {username='kudu', principal='kudu/127.30.247.62@KRBTEST.COM'} at 127.0.0.1:48942
I20260925 05:05:39.945741 19061 tablet_replica.cc:330] T 6e00b9c539444f47a724c658f4f1cffc P c590c18e610f46709ffb0b0139774e2f: stopping tablet replica
I20260925 05:05:39.946187 19061 raft_consensus.cc:2229] T 6e00b9c539444f47a724c658f4f1cffc P c590c18e610f46709ffb0b0139774e2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:39.946743 19061 raft_consensus.cc:2258] T 6e00b9c539444f47a724c658f4f1cffc P c590c18e610f46709ffb0b0139774e2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:39.946952 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 18622
I20260925 05:05:39.950142 19061 ts_tablet_manager.cc:1905] T 6e00b9c539444f47a724c658f4f1cffc P c590c18e610f46709ffb0b0139774e2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:39.958506 19061 ts_tablet_manager.cc:1918] T 6e00b9c539444f47a724c658f4f1cffc P c590c18e610f46709ffb0b0139774e2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:05:39.958717 19061 log.cc:1196] T 6e00b9c539444f47a724c658f4f1cffc P c590c18e610f46709ffb0b0139774e2f: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-1/wal/wals/6e00b9c539444f47a724c658f4f1cffc
I20260925 05:05:39.959297 19061 ts_tablet_manager.cc:1939] T 6e00b9c539444f47a724c658f4f1cffc P c590c18e610f46709ffb0b0139774e2f: Deleting consensus metadata
I20260925 05:05:39.961351 18546 catalog_manager.cc:4841] TS c590c18e610f46709ffb0b0139774e2f (127.30.247.2:35877): tablet 6e00b9c539444f47a724c658f4f1cffc (table test-table [id=ba6dab424fa6429fb00b5b43c2cc8248]) successfully deleted
I20260925 05:05:39.970824 19059 tablet_replica.cc:330] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527: stopping tablet replica
I20260925 05:05:39.971051 19060 tablet_replica.cc:330] T 6e00b9c539444f47a724c658f4f1cffc P 3433635518fb4a66af65fc2b3599f0f1: stopping tablet replica
I20260925 05:05:39.971449 19060 raft_consensus.cc:2229] T 6e00b9c539444f47a724c658f4f1cffc P 3433635518fb4a66af65fc2b3599f0f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:39.971462 19059 raft_consensus.cc:2229] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:05:39.972038 19060 raft_consensus.cc:2258] T 6e00b9c539444f47a724c658f4f1cffc P 3433635518fb4a66af65fc2b3599f0f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:39.972242 19059 raft_consensus.cc:2258] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:39.974260 19060 ts_tablet_manager.cc:1905] T 6e00b9c539444f47a724c658f4f1cffc P 3433635518fb4a66af65fc2b3599f0f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:39.974272 19059 ts_tablet_manager.cc:1905] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 05:05:39 dist-test-slave-5n86 krb5kdc[18516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.1@KRBTEST.COM
I20260925 05:05:39.981746 19059 ts_tablet_manager.cc:1918] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:05:39.981844 19060 ts_tablet_manager.cc:1918] T 6e00b9c539444f47a724c658f4f1cffc P 3433635518fb4a66af65fc2b3599f0f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:05:39.982013 19060 log.cc:1196] T 6e00b9c539444f47a724c658f4f1cffc P 3433635518fb4a66af65fc2b3599f0f1: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-2/wal/wals/6e00b9c539444f47a724c658f4f1cffc
I20260925 05:05:39.982048 19059 log.cc:1196] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312430481417-31708-0/minicluster-data/ts-0/wal/wals/6e00b9c539444f47a724c658f4f1cffc
I20260925 05:05:39.982637 19059 ts_tablet_manager.cc:1939] T 6e00b9c539444f47a724c658f4f1cffc P 4289df0eaeff4aee87f9e47842052527: Deleting consensus metadata
I20260925 05:05:39.982861 19060 ts_tablet_manager.cc:1939] T 6e00b9c539444f47a724c658f4f1cffc P 3433635518fb4a66af65fc2b3599f0f1: Deleting consensus metadata
I20260925 05:05:39.984558 18547 catalog_manager.cc:4841] TS 4289df0eaeff4aee87f9e47842052527 (127.30.247.1:46289): tablet 6e00b9c539444f47a724c658f4f1cffc (table test-table [id=ba6dab424fa6429fb00b5b43c2cc8248]) successfully deleted
I20260925 05:05:39.984549 18546 catalog_manager.cc:4841] TS 3433635518fb4a66af65fc2b3599f0f1 (127.30.247.3:41855): tablet 6e00b9c539444f47a724c658f4f1cffc (table test-table [id=ba6dab424fa6429fb00b5b43c2cc8248]) successfully deleted
W20260925 05:05:39.985420 18546 catalog_manager.cc:4808] TS 3433635518fb4a66af65fc2b3599f0f1 (127.30.247.3:41855): delete failed for tablet 6e00b9c539444f47a724c658f4f1cffc because the tablet was not found. No further retry: Not found: Tablet not found: 6e00b9c539444f47a724c658f4f1cffc
I20260925 05:05:39.987623 19031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:39.948566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50608 (local address 127.30.247.1:46289)
0925 05:05:39.948742 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:39.948751 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:39.967450 (+ 18699us) server_negotiation.cc:407] Connection header received
0925 05:05:39.967792 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:39.967802 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:39.967929 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:39.968032 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:39.970922 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.972202 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.974175 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.974897 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.974995 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:39.980465 (+ 5470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:39.980486 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:39.980504 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:39.980596 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:39.984748 (+ 4152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:39.985603 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:39.985612 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:39.985615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:39.985803 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:39.986224 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:39.986228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:39.986230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:39.986551 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:39.986764 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:39.987126 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 05:05:39.987296 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 05:05:39.988257 18745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:40.085798 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 18622
I20260925 05:05:40.104882 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 18756
Sep 25 05:05:40 dist-test-slave-5n86 krb5kdc[18516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.2@KRBTEST.COM
I20260925 05:05:40.119874 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:40.105618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44282 (local address 127.30.247.2:35877)
0925 05:05:40.105797 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:40.105801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:40.105820 (+ 19us) server_negotiation.cc:407] Connection header received
0925 05:05:40.106028 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:40.106032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:40.106095 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:40.106186 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:40.107581 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:40.108312 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:40.109782 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:40.110167 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:40.110257 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:40.114402 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:40.114416 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:40.114431 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:40.114472 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:40.117139 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:40.117823 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:40.117831 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:40.117833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:40.117953 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:40.118322 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:40.118327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:40.118330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:40.118702 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:40.119025 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:40.119317 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 05:05:40.119541 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 05:05:40.120608 18879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:40.209910 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 18756
I20260925 05:05:40.227113 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 18889
Sep 25 05:05:40 dist-test-slave-5n86 krb5kdc[18516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.3@KRBTEST.COM
I20260925 05:05:40.241888 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:40.227818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48954 (local address 127.30.247.3:41855)
0925 05:05:40.228027 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:40.228035 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:40.228056 (+ 21us) server_negotiation.cc:407] Connection header received
0925 05:05:40.228261 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:40.228267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:40.228341 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:40.228462 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:40.229714 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:40.230441 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:40.231715 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:40.232119 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:40.232247 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:40.236629 (+ 4382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:40.236644 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:40.236659 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:40.236691 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:40.239162 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:40.239849 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:40.239866 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:40.239868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:40.239993 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:40.240350 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:40.240355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:40.240360 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:40.240761 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:40.241053 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:40.241357 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 05:05:40.241586 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 05:05:40.242458 19012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:40.333446 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 18889
I20260925 05:05:40.350219 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 18529
I20260925 05:05:40.351405 18591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:40.462038 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 18529
2026-09-25T05:05:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4168 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T05:05:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:40Z Disabled control of system clock
I20260925 05:05:40.535696 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:38629
--webserver_interface=127.30.247.62
--webserver_port=0
--builtin_ntp_servers=127.30.247.20:44105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:38629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 05:05:40.858031 19070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:40.858297 19070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:40.858376 19070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:40.865593 19070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:40.865703 19070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:40.865741 19070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:40.865790 19070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:40.865837 19070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:40.874894 19070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:44105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:38629
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:38629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:40.876168 19070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:40.877915 19070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:40.889631 19075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:40.889673 19076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:40.889859 19078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:40.890635 19070 server_base.cc:1034] running on GCE node
I20260925 05:05:40.891366 19070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:40.893093 19070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:40.894304 19070 hybrid_clock.cc:648] HybridClock initialized: now 1790312740894245 us; error 58 us; skew 500 ppm
I20260925 05:05:40.894672 19070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:05:40.897099 19070 webserver.cc:460] Webserver started at http://127.30.247.62:45903/ using document root <none> and password file <none>
I20260925 05:05:40.897809 19070 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:40.897909 19070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:40.898248 19070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:40.901120 19070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "14a646fbb0934eaab7d85ef764b9cbf2"
format_stamp: "Formatted at 2026-09-25 05:05:40 on dist-test-slave-5n86"
server_key: "1177d37b594656c4817f8a0055050ac8"
server_key_iv: "30e94ae0acdb3c9994888bfd401ec844"
server_key_version: "encryptionkey@0"
I20260925 05:05:40.901914 19070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "14a646fbb0934eaab7d85ef764b9cbf2"
format_stamp: "Formatted at 2026-09-25 05:05:40 on dist-test-slave-5n86"
server_key: "1177d37b594656c4817f8a0055050ac8"
server_key_iv: "30e94ae0acdb3c9994888bfd401ec844"
server_key_version: "encryptionkey@0"
I20260925 05:05:40.906782 19070 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 05:05:40.910172 19084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:40.911415 19070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:05:40.911589 19070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "14a646fbb0934eaab7d85ef764b9cbf2"
format_stamp: "Formatted at 2026-09-25 05:05:40 on dist-test-slave-5n86"
server_key: "1177d37b594656c4817f8a0055050ac8"
server_key_iv: "30e94ae0acdb3c9994888bfd401ec844"
server_key_version: "encryptionkey@0"
I20260925 05:05:40.911796 19070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:40.933156 19070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:40.933933 19070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:40.934222 19070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:40.952343 19070 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:38629
I20260925 05:05:40.952363 19135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:38629 every 8 connection(s)
I20260925 05:05:40.953670 19070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:40.956231 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 19070
I20260925 05:05:40.956413 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:05:40.956879 31708 external_mini_cluster.cc:1428] Setting key 3b5df951736c7ceeab55a02a7f2f20e2
I20260925 05:05:40.957482 19136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:40.972395 19139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:40.958995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35686 (local address 127.30.247.62:38629)
0925 05:05:40.960186 (+ 1191us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:40.960201 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:40.960328 (+ 127us) server_negotiation.cc:407] Connection header received
0925 05:05:40.962228 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:40.962255 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:40.962853 (+ 598us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:40.963293 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:40.963825 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:40.965833 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:40.967163 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:40.968198 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:40.968385 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:40.969619 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:40.969630 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:40.969654 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:40.970370 (+ 716us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:40.970466 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:40.970492 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:40.970600 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:40.970839 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 05:05:40.971144 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":408,"threads_started":1}
I20260925 05:05:40.972395 19136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14a646fbb0934eaab7d85ef764b9cbf2: Bootstrap starting.
I20260925 05:05:40.975445 19136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14a646fbb0934eaab7d85ef764b9cbf2: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:40.976316 19136 log.cc:824] T 00000000000000000000000000000000 P 14a646fbb0934eaab7d85ef764b9cbf2: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:40.978816 19136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14a646fbb0934eaab7d85ef764b9cbf2: No bootstrap required, opened a new log
I20260925 05:05:40.987366 19136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14a646fbb0934eaab7d85ef764b9cbf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a646fbb0934eaab7d85ef764b9cbf2" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 38629 } }
I20260925 05:05:40.987787 19136 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14a646fbb0934eaab7d85ef764b9cbf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:40.987891 19136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14a646fbb0934eaab7d85ef764b9cbf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14a646fbb0934eaab7d85ef764b9cbf2, State: Initialized, Role: FOLLOWER
I20260925 05:05:40.988488 19136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14a646fbb0934eaab7d85ef764b9cbf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a646fbb0934eaab7d85ef764b9cbf2" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 38629 } }
I20260925 05:05:40.988826 19136 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14a646fbb0934eaab7d85ef764b9cbf2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:40.989006 19136 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14a646fbb0934eaab7d85ef764b9cbf2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:40.989188 19136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14a646fbb0934eaab7d85ef764b9cbf2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:40.991583 19136 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14a646fbb0934eaab7d85ef764b9cbf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a646fbb0934eaab7d85ef764b9cbf2" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 38629 } }
I20260925 05:05:40.992030 19136 leader_election.cc:302] T 00000000000000000000000000000000 P 14a646fbb0934eaab7d85ef764b9cbf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14a646fbb0934eaab7d85ef764b9cbf2; no voters:
I20260925 05:05:40.992643 19136 leader_election.cc:288] T 00000000000000000000000000000000 P 14a646fbb0934eaab7d85ef764b9cbf2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:40.992802 19141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14a646fbb0934eaab7d85ef764b9cbf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:40.993620 19141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14a646fbb0934eaab7d85ef764b9cbf2 [term 1 LEADER]: Becoming Leader. State: Replica: 14a646fbb0934eaab7d85ef764b9cbf2, State: Running, Role: LEADER
I20260925 05:05:40.994094 19141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14a646fbb0934eaab7d85ef764b9cbf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a646fbb0934eaab7d85ef764b9cbf2" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 38629 } }
I20260925 05:05:40.994812 19136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14a646fbb0934eaab7d85ef764b9cbf2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:40.997406 19143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14a646fbb0934eaab7d85ef764b9cbf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14a646fbb0934eaab7d85ef764b9cbf2. Latest consensus state: current_term: 1 leader_uuid: "14a646fbb0934eaab7d85ef764b9cbf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a646fbb0934eaab7d85ef764b9cbf2" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 38629 } } }
I20260925 05:05:40.997859 19143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14a646fbb0934eaab7d85ef764b9cbf2 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:41.000757 19142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14a646fbb0934eaab7d85ef764b9cbf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14a646fbb0934eaab7d85ef764b9cbf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a646fbb0934eaab7d85ef764b9cbf2" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 38629 } } }
I20260925 05:05:41.001132 19142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14a646fbb0934eaab7d85ef764b9cbf2 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:41.002216 19146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:41.007571 19146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:41.016507 19146 catalog_manager.cc:1347] Generated new cluster ID: 91ec1f84973c4b418851f37f6a688161
I20260925 05:05:41.016702 19146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:05:41.034160 19146 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:41.035437 19146 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:41.043483 19146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14a646fbb0934eaab7d85ef764b9cbf2: Generated new TSK 0
I20260925 05:05:41.044234 19146 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:05:41.053573 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.1:0
--local_ip_for_outbound_sockets=127.30.247.1
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:38629
--builtin_ntp_servers=127.30.247.20:44105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 05:05:41.393853 19160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:41.394109 19160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:41.394253 19160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:41.401630 19160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:41.401728 19160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:41.401901 19160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.1
I20260925 05:05:41.411242 19160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:44105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.247.1
--webserver_port=0
--tserver_master_addrs=127.30.247.62:38629
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.1
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:41.412599 19160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:41.414410 19160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:41.427891 19166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:41.427997 19165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:41.428392 19168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:41.429082 19160 server_base.cc:1034] running on GCE node
I20260925 05:05:41.429721 19160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:41.431416 19160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:41.432658 19160 hybrid_clock.cc:648] HybridClock initialized: now 1790312741432596 us; error 59 us; skew 500 ppm
I20260925 05:05:41.432998 19160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:05:41.435202 19160 webserver.cc:460] Webserver started at http://127.30.247.1:43135/ using document root <none> and password file <none>
I20260925 05:05:41.435905 19160 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:41.436028 19160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:41.436362 19160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:41.439316 19160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-0/data/instance:
uuid: "be34a1ae553c44aba6f17106208ca076"
format_stamp: "Formatted at 2026-09-25 05:05:41 on dist-test-slave-5n86"
server_key: "81accc4972c302b26da8a90b5308b814"
server_key_iv: "df0cb71ae5e21b0225efb2e45a6d1996"
server_key_version: "encryptionkey@0"
I20260925 05:05:41.440093 19160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance:
uuid: "be34a1ae553c44aba6f17106208ca076"
format_stamp: "Formatted at 2026-09-25 05:05:41 on dist-test-slave-5n86"
server_key: "81accc4972c302b26da8a90b5308b814"
server_key_iv: "df0cb71ae5e21b0225efb2e45a6d1996"
server_key_version: "encryptionkey@0"
I20260925 05:05:41.445022 19160 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260925 05:05:41.448323 19174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:41.449642 19160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:05:41.449803 19160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-0/wal
uuid: "be34a1ae553c44aba6f17106208ca076"
format_stamp: "Formatted at 2026-09-25 05:05:41 on dist-test-slave-5n86"
server_key: "81accc4972c302b26da8a90b5308b814"
server_key_iv: "df0cb71ae5e21b0225efb2e45a6d1996"
server_key_version: "encryptionkey@0"
I20260925 05:05:41.450021 19160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:41.462033 19160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:41.462744 19160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:41.463027 19160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:41.463886 19160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:41.465430 19160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:41.465549 19160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:41.465674 19160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:41.465764 19160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:41.497447 19160 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.1:43129
I20260925 05:05:41.497491 19286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.1:43129 every 8 connection(s)
I20260925 05:05:41.498750 19160 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-0/data/info.pb
I20260925 05:05:41.500859 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 19160
I20260925 05:05:41.501034 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-0/wal/instance
I20260925 05:05:41.501659 31708 external_mini_cluster.cc:1428] Setting key ab86e66358e92898478283217922923e
I20260925 05:05:41.504838 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.2:0
--local_ip_for_outbound_sockets=127.30.247.2
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:38629
--builtin_ntp_servers=127.30.247.20:44105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 05:05:41.514437 19289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:41.501736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.1:53689 (local address 127.30.247.62:38629)
0925 05:05:41.502574 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:41.502581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:41.503727 (+ 1146us) server_negotiation.cc:407] Connection header received
0925 05:05:41.506394 (+ 2667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:41.506399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:41.506474 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:41.506568 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:41.507953 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:41.508850 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:41.511323 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:41.511728 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:41.511846 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:41.513461 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:41.513466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:41.513469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:41.513519 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:41.513574 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:41.513595 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:41.513701 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:41.513912 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 05:05:41.514040 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":597,"threads_started":1}
I20260925 05:05:41.515260 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:41.502449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:38629 (local address 127.30.247.1:53689)
0925 05:05:41.503576 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:41.503620 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:41.506105 (+ 2485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:41.506827 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:41.506839 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:41.506957 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:41.507755 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:41.507782 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:41.508996 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:41.509004 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:41.511174 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:41.511192 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:41.511963 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:41.511973 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:41.512288 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:41.513079 (+ 791us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:05:41.513085 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:41.513137 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:05:41.513233 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:41.513709 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:41.513717 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:41.513723 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 05:05:41.513880 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 05:05:41.514150 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":414,"threads_started":1}
I20260925 05:05:41.516923 19287 heartbeater.cc:347] Connected to a master server at 127.30.247.62:38629
I20260925 05:05:41.517266 19287 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:41.518129 19287 heartbeater.cc:511] Master 127.30.247.62:38629 requested a full tablet report, sending...
I20260925 05:05:41.520409 19101 ts_manager.cc:194] Registered new tserver with Master: be34a1ae553c44aba6f17106208ca076 (127.30.247.1:43129)
I20260925 05:05:41.522610 19101 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.247.1:53689
W20260925 05:05:41.825886 19291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:41.826154 19291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:41.826237 19291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:41.833221 19291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:41.833305 19291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:41.833405 19291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.2
I20260925 05:05:41.842005 19291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:44105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.247.2
--webserver_port=0
--tserver_master_addrs=127.30.247.62:38629
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.2
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:41.843168 19291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:41.844933 19291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:41.857712 19296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:41.857752 19299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:41.857784 19297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:41.858671 19291 server_base.cc:1034] running on GCE node
I20260925 05:05:41.859230 19291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:41.860173 19291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:41.861434 19291 hybrid_clock.cc:648] HybridClock initialized: now 1790312741861394 us; error 60 us; skew 500 ppm
I20260925 05:05:41.861779 19291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:05:41.864032 19291 webserver.cc:460] Webserver started at http://127.30.247.2:44149/ using document root <none> and password file <none>
I20260925 05:05:41.864804 19291 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:41.864941 19291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:41.865255 19291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:41.868148 19291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-1/data/instance:
uuid: "91c285ee4f2946ec8e44260a58c55b72"
format_stamp: "Formatted at 2026-09-25 05:05:41 on dist-test-slave-5n86"
server_key: "98a99b353e90ad3dbd69bdb8c6a52510"
server_key_iv: "57e04cfb3726fa79507a6dbfb2324b42"
server_key_version: "encryptionkey@0"
I20260925 05:05:41.868978 19291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance:
uuid: "91c285ee4f2946ec8e44260a58c55b72"
format_stamp: "Formatted at 2026-09-25 05:05:41 on dist-test-slave-5n86"
server_key: "98a99b353e90ad3dbd69bdb8c6a52510"
server_key_iv: "57e04cfb3726fa79507a6dbfb2324b42"
server_key_version: "encryptionkey@0"
I20260925 05:05:41.873914 19291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 05:05:41.877316 19305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:41.878710 19291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:05:41.878881 19291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-1/wal
uuid: "91c285ee4f2946ec8e44260a58c55b72"
format_stamp: "Formatted at 2026-09-25 05:05:41 on dist-test-slave-5n86"
server_key: "98a99b353e90ad3dbd69bdb8c6a52510"
server_key_iv: "57e04cfb3726fa79507a6dbfb2324b42"
server_key_version: "encryptionkey@0"
I20260925 05:05:41.879089 19291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:41.903452 19291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:41.904289 19291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:41.904673 19291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:41.905584 19291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:41.906975 19291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:41.907094 19291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:41.907229 19291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:41.907322 19291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:41.939623 19291 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.2:44207
I20260925 05:05:41.939713 19417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.2:44207 every 8 connection(s)
I20260925 05:05:41.940990 19291 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-1/data/info.pb
I20260925 05:05:41.942906 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 19291
I20260925 05:05:41.943092 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-1/wal/instance
I20260925 05:05:41.943538 31708 external_mini_cluster.cc:1428] Setting key b283b11f14ba871797439792ec8f0f3a
I20260925 05:05:41.946630 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.247.3:0
--local_ip_for_outbound_sockets=127.30.247.3
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:38629
--builtin_ntp_servers=127.30.247.20:44105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 05:05:41.955607 19289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:41.943746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:33519 (local address 127.30.247.62:38629)
0925 05:05:41.943962 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:41.943969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:41.945418 (+ 1449us) server_negotiation.cc:407] Connection header received
0925 05:05:41.947496 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:41.947501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:41.947588 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:41.947689 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:41.949125 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:41.949838 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:41.952282 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:41.952962 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:41.953097 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:41.954726 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:41.954733 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:41.954738 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:41.954761 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:41.954800 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:41.954816 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:41.954920 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:41.955175 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 05:05:41.955288 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 05:05:41.956694 19420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:41.944342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:38629 (local address 127.30.247.2:33519)
0925 05:05:41.945264 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:41.945304 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:41.947228 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:41.947961 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:41.947976 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:41.948104 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:41.948948 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:41.948974 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:41.949993 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:41.950001 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:41.952125 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:41.952141 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:41.953189 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:41.953198 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:41.953528 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:41.954343 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:05:41.954349 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:41.954407 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:05:41.954512 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:41.954935 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:41.954949 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:41.954960 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 05:05:41.955143 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 05:05:41.955445 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":313,"threads_started":1}
I20260925 05:05:41.958532 19418 heartbeater.cc:347] Connected to a master server at 127.30.247.62:38629
I20260925 05:05:41.958860 19418 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:41.959808 19418 heartbeater.cc:511] Master 127.30.247.62:38629 requested a full tablet report, sending...
I20260925 05:05:41.961598 19101 ts_manager.cc:194] Registered new tserver with Master: 91c285ee4f2946ec8e44260a58c55b72 (127.30.247.2:44207)
I20260925 05:05:41.962780 19101 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.247.2:33519
W20260925 05:05:42.283188 19421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:42.283453 19421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:42.283536 19421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:42.290452 19421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:42.290534 19421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:42.290630 19421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.247.3
I20260925 05:05:42.299104 19421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:44105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.247.3
--webserver_port=0
--tserver_master_addrs=127.30.247.62:38629
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.247.3
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:42.300159 19421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:42.301834 19421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:42.314355 19426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:42.314311 19427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:42.314393 19429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:42.315587 19421 server_base.cc:1034] running on GCE node
I20260925 05:05:42.316171 19421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:42.317224 19421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:42.318420 19421 hybrid_clock.cc:648] HybridClock initialized: now 1790312742318389 us; error 32 us; skew 500 ppm
I20260925 05:05:42.318725 19421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 05:05:42.320935 19421 webserver.cc:460] Webserver started at http://127.30.247.3:38085/ using document root <none> and password file <none>
I20260925 05:05:42.321599 19421 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:42.321704 19421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:42.322028 19421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:42.324771 19421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-2/data/instance:
uuid: "bbf7c52ba7254b63b9d6818947f28713"
format_stamp: "Formatted at 2026-09-25 05:05:42 on dist-test-slave-5n86"
server_key: "402e6392ea1dc427d251e7f8b17876fb"
server_key_iv: "ea08522c8ac52ccab198a16c912f3b24"
server_key_version: "encryptionkey@0"
I20260925 05:05:42.325541 19421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance:
uuid: "bbf7c52ba7254b63b9d6818947f28713"
format_stamp: "Formatted at 2026-09-25 05:05:42 on dist-test-slave-5n86"
server_key: "402e6392ea1dc427d251e7f8b17876fb"
server_key_iv: "ea08522c8ac52ccab198a16c912f3b24"
server_key_version: "encryptionkey@0"
I20260925 05:05:42.330170 19421 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 05:05:42.333411 19435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:42.334642 19421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:05:42.334766 19421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-2/wal
uuid: "bbf7c52ba7254b63b9d6818947f28713"
format_stamp: "Formatted at 2026-09-25 05:05:42 on dist-test-slave-5n86"
server_key: "402e6392ea1dc427d251e7f8b17876fb"
server_key_iv: "ea08522c8ac52ccab198a16c912f3b24"
server_key_version: "encryptionkey@0"
I20260925 05:05:42.334933 19421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:42.348829 19421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:42.349495 19421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:42.349733 19421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:42.350554 19421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:42.351799 19421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:42.351897 19421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:42.351971 19421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:42.352025 19421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:42.386838 19421 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.3:42641
I20260925 05:05:42.386911 19547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.3:42641 every 8 connection(s)
I20260925 05:05:42.388083 19421 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-2/data/info.pb
I20260925 05:05:42.396459 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 19421
I20260925 05:05:42.396647 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-2/wal/instance
I20260925 05:05:42.397078 31708 external_mini_cluster.cc:1428] Setting key 6a0449b8c037ee0df87bcdd29b525cd1
I20260925 05:05:42.401024 19289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:42.390798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.3:32797 (local address 127.30.247.62:38629)
0925 05:05:42.390977 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:42.390983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:42.392286 (+ 1303us) server_negotiation.cc:407] Connection header received
0925 05:05:42.394042 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:42.394048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:42.394137 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:42.394248 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:42.395382 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.396281 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.398313 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.398819 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.398960 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:42.400224 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:42.400229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:42.400232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:42.400248 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:42.400278 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:42.400289 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:42.400381 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:42.400609 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 05:05:42.400705 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 05:05:42.401882 19550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:42.391306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:38629 (local address 127.30.247.3:32797)
0925 05:05:42.392141 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:42.392181 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:42.393775 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:42.394465 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:42.394494 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:42.394593 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:42.395235 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:42.395252 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.396453 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.396457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:42.398154 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:42.398171 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.399045 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.399052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:42.399265 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:42.399918 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:05:42.399921 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:42.399956 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:05:42.400037 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:42.400381 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:42.400392 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:42.400402 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 05:05:42.400590 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 05:05:42.400929 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":249,"threads_started":1}
I20260925 05:05:42.403113 19548 heartbeater.cc:347] Connected to a master server at 127.30.247.62:38629
I20260925 05:05:42.403347 19548 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:42.403983 19548 heartbeater.cc:511] Master 127.30.247.62:38629 requested a full tablet report, sending...
I20260925 05:05:42.405226 19101 ts_manager.cc:194] Registered new tserver with Master: bbf7c52ba7254b63b9d6818947f28713 (127.30.247.3:42641)
I20260925 05:05:42.405958 19101 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.247.3:32797
I20260925 05:05:42.411700 31708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:05:42.420936 19289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:42.415779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35698 (local address 127.30.247.62:38629)
0925 05:05:42.415916 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:42.415921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:42.416336 (+ 415us) server_negotiation.cc:407] Connection header received
0925 05:05:42.416457 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:42.416460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:42.416574 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:42.416657 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:42.417086 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.417725 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.418985 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.419348 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.419433 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:42.420352 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:42.420356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:42.420359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:42.420374 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:42.420402 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:42.420413 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:42.420487 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:42.420648 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 05:05:42.420719 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 05:05:42.424427 19101 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:05:42.431048 19101 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:05:42.465555 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:42.442686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:43129 (local address 127.0.0.1:33692)
0925 05:05:42.453509 (+ 10823us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:42.453562 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:42.453775 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:42.456717 (+ 2942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:42.456729 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:42.456770 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:42.457269 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:42.457281 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.460224 (+ 2943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.460229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:42.462667 (+ 2438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:42.462685 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.464595 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.464604 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:42.464831 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:42.464884 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 05:05:42.464998 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 05:05:42.465203 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10469,"thread_start_us":12143,"threads_started":1}
I20260925 05:05:42.465602 19561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:42.442898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33692 (local address 127.30.247.1:43129)
0925 05:05:42.455907 (+ 13009us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:42.455920 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:42.455957 (+ 37us) server_negotiation.cc:407] Connection header received
0925 05:05:42.456080 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:42.456091 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:42.456409 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:42.457517 (+ 1108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:42.457801 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.459387 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.462906 (+ 3519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.464329 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.464480 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:42.465030 (+ 550us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:42.465100 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 05:05:42.465261 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12799,"thread_start_us":12669,"threads_started":1}
I20260925 05:05:42.468500 19560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:42.442889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:42641 (local address 127.0.0.1:40474)
0925 05:05:42.455182 (+ 12293us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:42.455205 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:42.455350 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:42.459526 (+ 4176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:42.459534 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:42.459558 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:42.460014 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:42.460028 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.462110 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.462116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:42.465555 (+ 3439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:42.465571 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.467682 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.467691 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:42.467864 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:42.467890 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 05:05:42.468001 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 05:05:42.468122 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12112,"mutex_wait_us":50,"thread_start_us":11115,"threads_started":1}
I20260925 05:05:42.468576 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:42.443249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40474 (local address 127.30.247.3:42641)
0925 05:05:42.458777 (+ 15528us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:42.458785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:42.458818 (+ 33us) server_negotiation.cc:407] Connection header received
0925 05:05:42.458921 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:42.458929 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:42.459138 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:42.459304 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:42.460186 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.461944 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.465783 (+ 3839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.467322 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.467454 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:42.467714 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:42.468045 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 05:05:42.468198 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15344,"thread_start_us":16250,"threads_started":1}
I20260925 05:05:42.468618 19565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:42.455918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.2:44207 (local address 127.0.0.1:57680)
0925 05:05:42.461429 (+ 5511us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:42.461451 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:42.461621 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:42.462101 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:42.462105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:42.462133 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:42.462614 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:42.462627 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.463858 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.463863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:42.465889 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:42.465903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.467805 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.467812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:42.467997 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:42.468039 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 05:05:42.468128 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 05:05:42.468242 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4059,"mutex_wait_us":4,"thread_start_us":3885,"threads_started":1}
I20260925 05:05:42.469192 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:42.455710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57680 (local address 127.30.247.2:44207)
0925 05:05:42.461403 (+ 5693us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:42.461414 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:42.461556 (+ 142us) server_negotiation.cc:407] Connection header received
0925 05:05:42.461771 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:42.461777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:42.461932 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:42.462119 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:42.462787 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.463720 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.466103 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.467320 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.467445 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:42.467671 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:42.468738 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0925 05:05:42.468893 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5554,"thread_start_us":5466,"threads_started":1}
I20260925 05:05:42.469959 19222 tablet_service.cc:1467] Processing CreateTablet for tablet 5daa9f019e2e4fdea819635d1105cefc (DEFAULT_TABLE table=test-table [id=12e6457d6bb5480c9bfe633b46472779]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:42.471342 19222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5daa9f019e2e4fdea819635d1105cefc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:42.472373 19483 tablet_service.cc:1467] Processing CreateTablet for tablet 5daa9f019e2e4fdea819635d1105cefc (DEFAULT_TABLE table=test-table [id=12e6457d6bb5480c9bfe633b46472779]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:42.473217 19353 tablet_service.cc:1467] Processing CreateTablet for tablet 5daa9f019e2e4fdea819635d1105cefc (DEFAULT_TABLE table=test-table [id=12e6457d6bb5480c9bfe633b46472779]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:42.473878 19483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5daa9f019e2e4fdea819635d1105cefc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:42.474499 19353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5daa9f019e2e4fdea819635d1105cefc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:42.482388 19566 tablet_bootstrap.cc:492] T 5daa9f019e2e4fdea819635d1105cefc P be34a1ae553c44aba6f17106208ca076: Bootstrap starting.
I20260925 05:05:42.485982 19566 tablet_bootstrap.cc:654] T 5daa9f019e2e4fdea819635d1105cefc P be34a1ae553c44aba6f17106208ca076: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:42.487079 19566 log.cc:824] T 5daa9f019e2e4fdea819635d1105cefc P be34a1ae553c44aba6f17106208ca076: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:42.487965 19567 tablet_bootstrap.cc:492] T 5daa9f019e2e4fdea819635d1105cefc P bbf7c52ba7254b63b9d6818947f28713: Bootstrap starting.
I20260925 05:05:42.489722 19568 tablet_bootstrap.cc:492] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72: Bootstrap starting.
I20260925 05:05:42.491091 19566 tablet_bootstrap.cc:492] T 5daa9f019e2e4fdea819635d1105cefc P be34a1ae553c44aba6f17106208ca076: No bootstrap required, opened a new log
I20260925 05:05:42.491401 19566 ts_tablet_manager.cc:1397] T 5daa9f019e2e4fdea819635d1105cefc P be34a1ae553c44aba6f17106208ca076: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 05:05:42.491943 19567 tablet_bootstrap.cc:654] T 5daa9f019e2e4fdea819635d1105cefc P bbf7c52ba7254b63b9d6818947f28713: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:42.492720 19568 tablet_bootstrap.cc:654] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:42.493157 19567 log.cc:824] T 5daa9f019e2e4fdea819635d1105cefc P bbf7c52ba7254b63b9d6818947f28713: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:42.493681 19568 log.cc:824] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:42.496183 19567 tablet_bootstrap.cc:492] T 5daa9f019e2e4fdea819635d1105cefc P bbf7c52ba7254b63b9d6818947f28713: No bootstrap required, opened a new log
I20260925 05:05:42.496454 19567 ts_tablet_manager.cc:1397] T 5daa9f019e2e4fdea819635d1105cefc P bbf7c52ba7254b63b9d6818947f28713: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 05:05:42.496682 19568 tablet_bootstrap.cc:492] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72: No bootstrap required, opened a new log
I20260925 05:05:42.496934 19568 ts_tablet_manager.cc:1397] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260925 05:05:42.502679 19566 raft_consensus.cc:348] T 5daa9f019e2e4fdea819635d1105cefc P be34a1ae553c44aba6f17106208ca076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be34a1ae553c44aba6f17106208ca076" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 43129 } } peers { permanent_uuid: "bbf7c52ba7254b63b9d6818947f28713" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 42641 } } peers { permanent_uuid: "91c285ee4f2946ec8e44260a58c55b72" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44207 } }
I20260925 05:05:42.503127 19566 raft_consensus.cc:374] T 5daa9f019e2e4fdea819635d1105cefc P be34a1ae553c44aba6f17106208ca076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:42.503238 19566 raft_consensus.cc:729] T 5daa9f019e2e4fdea819635d1105cefc P be34a1ae553c44aba6f17106208ca076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be34a1ae553c44aba6f17106208ca076, State: Initialized, Role: FOLLOWER
I20260925 05:05:42.503880 19566 consensus_queue.cc:260] T 5daa9f019e2e4fdea819635d1105cefc P be34a1ae553c44aba6f17106208ca076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be34a1ae553c44aba6f17106208ca076" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 43129 } } peers { permanent_uuid: "bbf7c52ba7254b63b9d6818947f28713" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 42641 } } peers { permanent_uuid: "91c285ee4f2946ec8e44260a58c55b72" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44207 } }
I20260925 05:05:42.507249 19568 raft_consensus.cc:348] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be34a1ae553c44aba6f17106208ca076" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 43129 } } peers { permanent_uuid: "bbf7c52ba7254b63b9d6818947f28713" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 42641 } } peers { permanent_uuid: "91c285ee4f2946ec8e44260a58c55b72" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44207 } }
I20260925 05:05:42.507249 19567 raft_consensus.cc:348] T 5daa9f019e2e4fdea819635d1105cefc P bbf7c52ba7254b63b9d6818947f28713 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be34a1ae553c44aba6f17106208ca076" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 43129 } } peers { permanent_uuid: "bbf7c52ba7254b63b9d6818947f28713" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 42641 } } peers { permanent_uuid: "91c285ee4f2946ec8e44260a58c55b72" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44207 } }
I20260925 05:05:42.507650 19567 raft_consensus.cc:374] T 5daa9f019e2e4fdea819635d1105cefc P bbf7c52ba7254b63b9d6818947f28713 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:42.507705 19568 raft_consensus.cc:374] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:42.507814 19567 raft_consensus.cc:729] T 5daa9f019e2e4fdea819635d1105cefc P bbf7c52ba7254b63b9d6818947f28713 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf7c52ba7254b63b9d6818947f28713, State: Initialized, Role: FOLLOWER
I20260925 05:05:42.507827 19568 raft_consensus.cc:729] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c285ee4f2946ec8e44260a58c55b72, State: Initialized, Role: FOLLOWER
I20260925 05:05:42.508517 19568 consensus_queue.cc:260] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be34a1ae553c44aba6f17106208ca076" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 43129 } } peers { permanent_uuid: "bbf7c52ba7254b63b9d6818947f28713" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 42641 } } peers { permanent_uuid: "91c285ee4f2946ec8e44260a58c55b72" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44207 } }
I20260925 05:05:42.508545 19567 consensus_queue.cc:260] T 5daa9f019e2e4fdea819635d1105cefc P bbf7c52ba7254b63b9d6818947f28713 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be34a1ae553c44aba6f17106208ca076" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 43129 } } peers { permanent_uuid: "bbf7c52ba7254b63b9d6818947f28713" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 42641 } } peers { permanent_uuid: "91c285ee4f2946ec8e44260a58c55b72" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44207 } }
I20260925 05:05:42.517042 19567 ts_tablet_manager.cc:1428] T 5daa9f019e2e4fdea819635d1105cefc P bbf7c52ba7254b63b9d6818947f28713: Time spent starting tablet: real 0.020s user 0.012s sys 0.007s
I20260925 05:05:42.517937 19418 heartbeater.cc:503] Master 127.30.247.62:38629 was elected leader, sending a full tablet report...
I20260925 05:05:42.518522 19568 ts_tablet_manager.cc:1428] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72: Time spent starting tablet: real 0.021s user 0.009s sys 0.011s
I20260925 05:05:42.519428 19548 heartbeater.cc:503] Master 127.30.247.62:38629 was elected leader, sending a full tablet report...
I20260925 05:05:42.519757 19287 heartbeater.cc:503] Master 127.30.247.62:38629 was elected leader, sending a full tablet report...
I20260925 05:05:42.520330 19566 ts_tablet_manager.cc:1428] T 5daa9f019e2e4fdea819635d1105cefc P be34a1ae553c44aba6f17106208ca076: Time spent starting tablet: real 0.029s user 0.017s sys 0.008s
I20260925 05:05:42.589495 19573 raft_consensus.cc:482] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:05:42.589818 19573 raft_consensus.cc:504] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be34a1ae553c44aba6f17106208ca076" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 43129 } } peers { permanent_uuid: "bbf7c52ba7254b63b9d6818947f28713" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 42641 } } peers { permanent_uuid: "91c285ee4f2946ec8e44260a58c55b72" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44207 } }
I20260925 05:05:42.591573 19573 leader_election.cc:288] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be34a1ae553c44aba6f17106208ca076 (127.30.247.1:43129), bbf7c52ba7254b63b9d6818947f28713 (127.30.247.3:42641)
I20260925 05:05:42.598985 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:42.592136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:50205 (local address 127.30.247.3:42641)
0925 05:05:42.592352 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:42.592358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:42.593013 (+ 655us) server_negotiation.cc:407] Connection header received
0925 05:05:42.593212 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:42.593215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:42.593272 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:42.593363 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:42.594010 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.594697 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.597081 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.598065 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.598244 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:42.598307 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:42.598636 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 05:05:42.598754 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 05:05:42.598989 19561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:42.592083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.247.2:43231 (local address 127.30.247.1:43129)
0925 05:05:42.592290 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:42.592297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:42.592906 (+ 609us) server_negotiation.cc:407] Connection header received
0925 05:05:42.593138 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:42.593142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:42.593212 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:42.593311 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:42.593896 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.594602 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.597151 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.597985 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.598080 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:42.598126 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:42.598609 (+ 483us) server_negotiation.cc:299] Negotiation successful
0925 05:05:42.598736 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 05:05:42.599081 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:42.592135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.3:42641 (local address 127.30.247.2:50205)
0925 05:05:42.592890 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:42.592904 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:42.593069 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:42.593388 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:42.593392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:42.593420 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:42.593834 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:42.593843 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.594837 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.594843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:42.596862 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:42.596881 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.598258 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.598265 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:42.598441 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:42.598484 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 05:05:42.598588 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 05:05:42.598706 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":146,"thread_start_us":475,"threads_started":1}
I20260925 05:05:42.599160 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:42.591914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.1:43129 (local address 127.30.247.2:43231)
0925 05:05:42.592802 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:42.592825 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:42.592978 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:42.593326 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:42.593331 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:42.593364 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:42.593733 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:42.593743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.594735 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.594743 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:42.596911 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:42.596933 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.598289 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.598294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:42.598428 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:42.598477 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 05:05:42.598548 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 05:05:42.598693 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":197,"spinlock_wait_cycles":22272,"thread_start_us":446,"threads_started":1}
I20260925 05:05:42.600134 19242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5daa9f019e2e4fdea819635d1105cefc" candidate_uuid: "91c285ee4f2946ec8e44260a58c55b72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be34a1ae553c44aba6f17106208ca076" is_pre_election: true
I20260925 05:05:42.600301 19503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5daa9f019e2e4fdea819635d1105cefc" candidate_uuid: "91c285ee4f2946ec8e44260a58c55b72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7c52ba7254b63b9d6818947f28713" is_pre_election: true
I20260925 05:05:42.600724 19242 raft_consensus.cc:2454] T 5daa9f019e2e4fdea819635d1105cefc P be34a1ae553c44aba6f17106208ca076 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91c285ee4f2946ec8e44260a58c55b72 in term 0.
I20260925 05:05:42.600903 19503 raft_consensus.cc:2454] T 5daa9f019e2e4fdea819635d1105cefc P bbf7c52ba7254b63b9d6818947f28713 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91c285ee4f2946ec8e44260a58c55b72 in term 0.
I20260925 05:05:42.601435 19308 leader_election.cc:302] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c285ee4f2946ec8e44260a58c55b72, be34a1ae553c44aba6f17106208ca076; no voters:
I20260925 05:05:42.601804 19573 raft_consensus.cc:2781] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:05:42.601969 19573 raft_consensus.cc:482] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:05:42.602090 19573 raft_consensus.cc:3037] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:42.604472 19573 raft_consensus.cc:504] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be34a1ae553c44aba6f17106208ca076" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 43129 } } peers { permanent_uuid: "bbf7c52ba7254b63b9d6818947f28713" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 42641 } } peers { permanent_uuid: "91c285ee4f2946ec8e44260a58c55b72" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44207 } }
I20260925 05:05:42.605057 19573 leader_election.cc:288] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72 [CANDIDATE]: Term 1 election: Requested vote from peers be34a1ae553c44aba6f17106208ca076 (127.30.247.1:43129), bbf7c52ba7254b63b9d6818947f28713 (127.30.247.3:42641)
I20260925 05:05:42.605499 19503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5daa9f019e2e4fdea819635d1105cefc" candidate_uuid: "91c285ee4f2946ec8e44260a58c55b72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf7c52ba7254b63b9d6818947f28713"
I20260925 05:05:42.605504 19242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5daa9f019e2e4fdea819635d1105cefc" candidate_uuid: "91c285ee4f2946ec8e44260a58c55b72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be34a1ae553c44aba6f17106208ca076"
I20260925 05:05:42.605682 19503 raft_consensus.cc:3037] T 5daa9f019e2e4fdea819635d1105cefc P bbf7c52ba7254b63b9d6818947f28713 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:42.605700 19242 raft_consensus.cc:3037] T 5daa9f019e2e4fdea819635d1105cefc P be34a1ae553c44aba6f17106208ca076 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:42.607787 19503 raft_consensus.cc:2454] T 5daa9f019e2e4fdea819635d1105cefc P bbf7c52ba7254b63b9d6818947f28713 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c285ee4f2946ec8e44260a58c55b72 in term 1.
I20260925 05:05:42.607810 19242 raft_consensus.cc:2454] T 5daa9f019e2e4fdea819635d1105cefc P be34a1ae553c44aba6f17106208ca076 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c285ee4f2946ec8e44260a58c55b72 in term 1.
I20260925 05:05:42.608175 19308 leader_election.cc:302] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 91c285ee4f2946ec8e44260a58c55b72, bbf7c52ba7254b63b9d6818947f28713, be34a1ae553c44aba6f17106208ca076; no voters:
I20260925 05:05:42.608523 19573 raft_consensus.cc:2781] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:42.608989 19573 raft_consensus.cc:686] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72 [term 1 LEADER]: Becoming Leader. State: Replica: 91c285ee4f2946ec8e44260a58c55b72, State: Running, Role: LEADER
I20260925 05:05:42.609436 19573 consensus_queue.cc:237] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be34a1ae553c44aba6f17106208ca076" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 43129 } } peers { permanent_uuid: "bbf7c52ba7254b63b9d6818947f28713" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 42641 } } peers { permanent_uuid: "91c285ee4f2946ec8e44260a58c55b72" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44207 } }
I20260925 05:05:42.614068 19099 catalog_manager.cc:5495] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91c285ee4f2946ec8e44260a58c55b72 (127.30.247.2). New cstate: current_term: 1 leader_uuid: "91c285ee4f2946ec8e44260a58c55b72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be34a1ae553c44aba6f17106208ca076" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 43129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbf7c52ba7254b63b9d6818947f28713" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 42641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91c285ee4f2946ec8e44260a58c55b72" member_type: VOTER last_known_addr { host: "127.30.247.2" port: 44207 } health_report { overall_health: HEALTHY } } }
W20260925 05:05:42.639930 19549 tablet.cc:2369] T 5daa9f019e2e4fdea819635d1105cefc P bbf7c52ba7254b63b9d6818947f28713: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:05:42.660923 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:42.655894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57684 (local address 127.30.247.2:44207)
0925 05:05:42.656080 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:42.656085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:42.656102 (+ 17us) server_negotiation.cc:407] Connection header received
0925 05:05:42.656256 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:42.656260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:42.656347 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:42.656494 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:05:42.656912 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.657606 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.658924 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.659341 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.659432 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:42.659856 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:05:42.660335 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:05:42.660448 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:42.660659 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 05:05:42.660721 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 05:05:42.668661 19242 raft_consensus.cc:1261] T 5daa9f019e2e4fdea819635d1105cefc P be34a1ae553c44aba6f17106208ca076 [term 1 FOLLOWER]: Refusing update from remote peer 91c285ee4f2946ec8e44260a58c55b72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:42.668889 19503 raft_consensus.cc:1261] T 5daa9f019e2e4fdea819635d1105cefc P bbf7c52ba7254b63b9d6818947f28713 [term 1 FOLLOWER]: Refusing update from remote peer 91c285ee4f2946ec8e44260a58c55b72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:42.669649 19577 consensus_queue.cc:1035] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be34a1ae553c44aba6f17106208ca076" member_type: VOTER last_known_addr { host: "127.30.247.1" port: 43129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:42.669965 19573 consensus_queue.cc:1035] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf7c52ba7254b63b9d6818947f28713" member_type: VOTER last_known_addr { host: "127.30.247.3" port: 42641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:42.679884 19582 mvcc.cc:205] Tried to move back new op lower bound from 7333120993961656320 to 7333120993732112384. Current Snapshot: MvccSnapshot[applied={T|T < 7333120993961656320}]
I20260925 05:05:42.694571 19100 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:42.694888 19100 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:42.698807 19100 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14a646fbb0934eaab7d85ef764b9cbf2: Sending DeleteTablet for 3 replicas of tablet 5daa9f019e2e4fdea819635d1105cefc
I20260925 05:05:42.700069 19483 tablet_service.cc:1514] Processing DeleteTablet for tablet 5daa9f019e2e4fdea819635d1105cefc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:42 UTC) from {username='slave'} at 127.0.0.1:40474
I20260925 05:05:42.700116 19353 tablet_service.cc:1514] Processing DeleteTablet for tablet 5daa9f019e2e4fdea819635d1105cefc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:42 UTC) from {username='slave'} at 127.0.0.1:57680
I20260925 05:05:42.700110 19222 tablet_service.cc:1514] Processing DeleteTablet for tablet 5daa9f019e2e4fdea819635d1105cefc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:42 UTC) from {username='slave'} at 127.0.0.1:33692
I20260925 05:05:42.703012 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 19160
I20260925 05:05:42.716660 19591 tablet_replica.cc:330] T 5daa9f019e2e4fdea819635d1105cefc P be34a1ae553c44aba6f17106208ca076: stopping tablet replica
I20260925 05:05:42.717096 19592 tablet_replica.cc:330] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72: stopping tablet replica
I20260925 05:05:42.717303 19591 raft_consensus.cc:2229] T 5daa9f019e2e4fdea819635d1105cefc P be34a1ae553c44aba6f17106208ca076 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:42.717514 19592 raft_consensus.cc:2229] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:05:42.717972 19591 raft_consensus.cc:2258] T 5daa9f019e2e4fdea819635d1105cefc P be34a1ae553c44aba6f17106208ca076 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:42.718230 19592 raft_consensus.cc:2258] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:42.719961 19590 tablet_replica.cc:330] T 5daa9f019e2e4fdea819635d1105cefc P bbf7c52ba7254b63b9d6818947f28713: stopping tablet replica
I20260925 05:05:42.720433 19591 ts_tablet_manager.cc:1905] T 5daa9f019e2e4fdea819635d1105cefc P be34a1ae553c44aba6f17106208ca076: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:42.720389 19590 raft_consensus.cc:2229] T 5daa9f019e2e4fdea819635d1105cefc P bbf7c52ba7254b63b9d6818947f28713 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:42.720908 19590 raft_consensus.cc:2258] T 5daa9f019e2e4fdea819635d1105cefc P bbf7c52ba7254b63b9d6818947f28713 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:42.724155 19590 ts_tablet_manager.cc:1905] T 5daa9f019e2e4fdea819635d1105cefc P bbf7c52ba7254b63b9d6818947f28713: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:42.724290 19592 ts_tablet_manager.cc:1905] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:42.727864 19591 ts_tablet_manager.cc:1918] T 5daa9f019e2e4fdea819635d1105cefc P be34a1ae553c44aba6f17106208ca076: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:05:42.728060 19591 log.cc:1196] T 5daa9f019e2e4fdea819635d1105cefc P be34a1ae553c44aba6f17106208ca076: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-0/wal/wals/5daa9f019e2e4fdea819635d1105cefc
I20260925 05:05:42.728693 19591 ts_tablet_manager.cc:1939] T 5daa9f019e2e4fdea819635d1105cefc P be34a1ae553c44aba6f17106208ca076: Deleting consensus metadata
I20260925 05:05:42.731118 19590 ts_tablet_manager.cc:1918] T 5daa9f019e2e4fdea819635d1105cefc P bbf7c52ba7254b63b9d6818947f28713: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:05:42.731290 19590 log.cc:1196] T 5daa9f019e2e4fdea819635d1105cefc P bbf7c52ba7254b63b9d6818947f28713: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-2/wal/wals/5daa9f019e2e4fdea819635d1105cefc
I20260925 05:05:42.731455 19087 catalog_manager.cc:4841] TS be34a1ae553c44aba6f17106208ca076 (127.30.247.1:43129): tablet 5daa9f019e2e4fdea819635d1105cefc (table test-table [id=12e6457d6bb5480c9bfe633b46472779]) successfully deleted
I20260925 05:05:42.731796 19590 ts_tablet_manager.cc:1939] T 5daa9f019e2e4fdea819635d1105cefc P bbf7c52ba7254b63b9d6818947f28713: Deleting consensus metadata
I20260925 05:05:42.734195 19085 catalog_manager.cc:4841] TS bbf7c52ba7254b63b9d6818947f28713 (127.30.247.3:42641): tablet 5daa9f019e2e4fdea819635d1105cefc (table test-table [id=12e6457d6bb5480c9bfe633b46472779]) successfully deleted
I20260925 05:05:42.734540 19561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:42.704364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33696 (local address 127.30.247.1:43129)
0925 05:05:42.704593 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:42.704599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:42.719271 (+ 14672us) server_negotiation.cc:407] Connection header received
0925 05:05:42.719505 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:42.719511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:42.719628 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:42.719740 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:42.720443 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.721609 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.730363 (+ 8754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.731009 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.731109 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:42.733102 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:42.733113 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:42.733130 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:42.733718 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:42.733768 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:42.733787 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:42.733927 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:42.734082 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 05:05:42.734197 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 05:05:42.735574 19282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:42.736083 19592 ts_tablet_manager.cc:1918] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:05:42.736260 19592 log.cc:1196] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72: Deleting WAL directory at /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312430481417-31708-0/minicluster-data/ts-1/wal/wals/5daa9f019e2e4fdea819635d1105cefc
I20260925 05:05:42.736856 19592 ts_tablet_manager.cc:1939] T 5daa9f019e2e4fdea819635d1105cefc P 91c285ee4f2946ec8e44260a58c55b72: Deleting consensus metadata
I20260925 05:05:42.738566 19087 catalog_manager.cc:4841] TS 91c285ee4f2946ec8e44260a58c55b72 (127.30.247.2:44207): tablet 5daa9f019e2e4fdea819635d1105cefc (table test-table [id=12e6457d6bb5480c9bfe633b46472779]) successfully deleted
I20260925 05:05:42.829952 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 19160
I20260925 05:05:42.854889 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 19291
I20260925 05:05:42.862066 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:42.855695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57700 (local address 127.30.247.2:44207)
0925 05:05:42.855900 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:42.855906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:42.855926 (+ 20us) server_negotiation.cc:407] Connection header received
0925 05:05:42.856134 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:42.856138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:42.856220 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:42.856339 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:42.856786 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.857585 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.858873 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.859314 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.859469 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:42.860523 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:42.860589 (+ 66us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:42.860615 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:42.861261 (+ 646us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:42.861312 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:42.861331 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:42.861469 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:42.861609 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 05:05:42.861708 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 05:05:42.862931 19413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:42.976368 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 19291
I20260925 05:05:42.995119 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 19421
I20260925 05:05:43.004767 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:42.995847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40490 (local address 127.30.247.3:42641)
0925 05:05:42.996066 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:42.996071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:42.996154 (+ 83us) server_negotiation.cc:407] Connection header received
0925 05:05:42.996344 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:42.996349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:42.996439 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:42.996524 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:42.996944 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.997645 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.998883 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:42.999254 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:42.999344 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:43.003503 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:43.003512 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:43.003526 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:43.004070 (+ 544us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:43.004118 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:43.004138 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:43.004222 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:43.004344 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 05:05:43.004426 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 05:05:43.005416 19543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:43.092334 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 19421
I20260925 05:05:43.116240 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 19070
I20260925 05:05:43.117493 19131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:43.238265 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 19070
2026-09-25T05:05:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2775 ms)
[----------] 6 tests from EncryptionPolicyTest (20759 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:05:43 dist-test-slave-5n86 krb5kdc[19600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:05:43 dist-test-slave-5n86 krb5kdc[19600](info): set up 2 sockets
Sep 25 05:05:43 dist-test-slave-5n86 krb5kdc[19600](info): commencing operation
krb5kdc: starting...
W20260925 05:05:44.349411 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:05:44 dist-test-slave-5n86 krb5kdc[19600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:05:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:05:44.600173 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38115
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:44.919903 19613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:44.920156 19613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:44.920320 19613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:44.927470 19613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:44.927575 19613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:44.927675 19613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:44.927793 19613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:44.927871 19613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:44.937265 19613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38115
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:44.938522 19613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:44.939824 19613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:44.951301 19618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:44.951264 19619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:44.951334 19621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:44.952073 19613 server_base.cc:1034] running on GCE node
I20260925 05:05:44.952858 19613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:44.954172 19613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:44.955369 19613 hybrid_clock.cc:648] HybridClock initialized: now 1790312744955315 us; error 49 us; skew 500 ppm
I20260925 05:05:44.955710 19613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:05:44 dist-test-slave-5n86 krb5kdc[19600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:44.958848 19613 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:05:44.960143 19613 webserver.cc:460] Webserver started at http://127.0.0.1:46757/ using document root <none> and password file <none>
I20260925 05:05:44.960796 19613 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:44.960892 19613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:44.961133 19613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:44.963907 19613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "62acb00642214161a0216097aa9a83f9"
format_stamp: "Formatted at 2026-09-25 05:05:44 on dist-test-slave-5n86"
server_key: "66c7f10cce870fbcd922cc5349b5f6d6"
server_key_iv: "21e76887635fd1e40f55845479ca5cf4"
server_key_version: "encryptionkey@0"
I20260925 05:05:44.964670 19613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "62acb00642214161a0216097aa9a83f9"
format_stamp: "Formatted at 2026-09-25 05:05:44 on dist-test-slave-5n86"
server_key: "66c7f10cce870fbcd922cc5349b5f6d6"
server_key_iv: "21e76887635fd1e40f55845479ca5cf4"
server_key_version: "encryptionkey@0"
I20260925 05:05:44.969448 19613 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 05:05:44.972759 19628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:44.973980 19613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:05:44.974121 19613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "62acb00642214161a0216097aa9a83f9"
format_stamp: "Formatted at 2026-09-25 05:05:44 on dist-test-slave-5n86"
server_key: "66c7f10cce870fbcd922cc5349b5f6d6"
server_key_iv: "21e76887635fd1e40f55845479ca5cf4"
server_key_version: "encryptionkey@0"
I20260925 05:05:44.974269 19613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:44.999203 19613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:45.002918 19613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:45.003147 19613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:45.021066 19613 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38115
I20260925 05:05:45.021147 19679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38115 every 8 connection(s)
I20260925 05:05:45.022334 19613 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:45.023010 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 19613
I20260925 05:05:45.023214 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:05:45.024006 31708 external_mini_cluster.cc:1428] Setting key 4ceddb26e4ad2596f308e679639fdcfc
I20260925 05:05:45.026554 19680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 05:05:45 dist-test-slave-5n86 krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 05:05:45.039512 19680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62acb00642214161a0216097aa9a83f9: Bootstrap starting.
I20260925 05:05:45.043558 19680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62acb00642214161a0216097aa9a83f9: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:45.044821 19680 log.cc:824] T 00000000000000000000000000000000 P 62acb00642214161a0216097aa9a83f9: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:45.047880 19680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62acb00642214161a0216097aa9a83f9: No bootstrap required, opened a new log
I20260925 05:05:45.048724 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:45.025920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34239 (local address 127.0.0.1:38115)
0925 05:05:45.026795 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:45.026806 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:45.026850 (+ 44us) server_negotiation.cc:407] Connection header received
0925 05:05:45.028390 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:45.028409 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:45.028840 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:45.029167 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:45.031036 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:45.032614 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:45.034446 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:45.035097 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:45.035246 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:45.040107 (+ 4861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:45.040116 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:45.040133 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:45.040178 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:45.044167 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:45.045072 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:45.045080 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:45.045086 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:45.045278 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:45.045793 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:45.045804 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:45.045809 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:45.046386 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:45.046620 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:45.047193 (+ 573us) server_negotiation.cc:299] Negotiation successful
0925 05:05:45.047577 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":391,"threads_started":1}
I20260925 05:05:45.052748 31708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:05:45.057442 19680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62acb00642214161a0216097aa9a83f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62acb00642214161a0216097aa9a83f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38115 } }
I20260925 05:05:45.057889 19680 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62acb00642214161a0216097aa9a83f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:45.057999 19680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62acb00642214161a0216097aa9a83f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62acb00642214161a0216097aa9a83f9, State: Initialized, Role: FOLLOWER
I20260925 05:05:45.058635 19680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62acb00642214161a0216097aa9a83f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62acb00642214161a0216097aa9a83f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38115 } }
I20260925 05:05:45.058957 19680 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62acb00642214161a0216097aa9a83f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:45.059118 19680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62acb00642214161a0216097aa9a83f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:45.059307 19680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62acb00642214161a0216097aa9a83f9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:45.061511 19680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62acb00642214161a0216097aa9a83f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62acb00642214161a0216097aa9a83f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38115 } }
I20260925 05:05:45.061987 19680 leader_election.cc:302] T 00000000000000000000000000000000 P 62acb00642214161a0216097aa9a83f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62acb00642214161a0216097aa9a83f9; no voters:
I20260925 05:05:45.062649 19680 leader_election.cc:288] T 00000000000000000000000000000000 P 62acb00642214161a0216097aa9a83f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:45.062736 19690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62acb00642214161a0216097aa9a83f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:45.065083 19680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62acb00642214161a0216097aa9a83f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:45.068185 19690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62acb00642214161a0216097aa9a83f9 [term 1 LEADER]: Becoming Leader. State: Replica: 62acb00642214161a0216097aa9a83f9, State: Running, Role: LEADER
I20260925 05:05:45.068888 19690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62acb00642214161a0216097aa9a83f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62acb00642214161a0216097aa9a83f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38115 } }
I20260925 05:05:45.072072 19691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62acb00642214161a0216097aa9a83f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62acb00642214161a0216097aa9a83f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62acb00642214161a0216097aa9a83f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38115 } } }
I20260925 05:05:45.072450 19691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62acb00642214161a0216097aa9a83f9 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:45.072810 19692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62acb00642214161a0216097aa9a83f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62acb00642214161a0216097aa9a83f9. Latest consensus state: current_term: 1 leader_uuid: "62acb00642214161a0216097aa9a83f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62acb00642214161a0216097aa9a83f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38115 } } }
I20260925 05:05:45.073041 19692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62acb00642214161a0216097aa9a83f9 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:45.076855 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:45.058621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60231 (local address 127.0.0.1:38115)
0925 05:05:45.058809 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:45.058815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:45.059311 (+ 496us) server_negotiation.cc:407] Connection header received
0925 05:05:45.059482 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:45.059487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:45.059557 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:45.059664 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:45.061048 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:45.061895 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:45.063418 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:45.064047 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:45.064122 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:45.066781 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:45.066789 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:45.066796 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:45.066848 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:45.071576 (+ 4728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:45.073014 (+ 1438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:45.073022 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:45.073032 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:45.073464 (+ 432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:45.074347 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:45.074356 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:45.074362 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:45.074817 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:45.075192 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:45.076078 (+ 886us) server_negotiation.cc:299] Negotiation successful
0925 05:05:45.076314 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 05:05:45.085471 19702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:45.090507 19702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:45.098779 19702 catalog_manager.cc:1347] Generated new cluster ID: 90063ec0a7bc4167be3a3a7ae959e7a9
I20260925 05:05:45.098893 19702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:05:45.112944 19702 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:45.114660 19702 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:45.122316 19702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62acb00642214161a0216097aa9a83f9: Generated new TSK 0
I20260925 05:05:45.123221 19702 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:05:45.131032 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 19613
I20260925 05:05:45.132977 19675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:45.227366 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 19613
W20260925 05:05:45.244338 19598 connection.cc:535] client connection to 127.0.0.1:38115 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38115): Connection reset by peer (error 104)
2026-09-25T05:05:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1984 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:05:45 dist-test-slave-5n86 krb5kdc[19712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:05:45 dist-test-slave-5n86 krb5kdc[19712](info): set up 2 sockets
Sep 25 05:05:45 dist-test-slave-5n86 krb5kdc[19712](info): commencing operation
krb5kdc: starting...
W20260925 05:05:46.325127 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:05:46 dist-test-slave-5n86 krb5kdc[19712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:05:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:05:46.564127 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39209
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:46.879333 19725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:46.879681 19725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:46.879837 19725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:46.887204 19725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:46.887358 19725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:46.887465 19725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:46.887580 19725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:46.887665 19725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:46.897703 19725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39209
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:46.899070 19725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:46.900408 19725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:46.912052 19730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:46.912141 19731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:46.912161 19733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:46.912951 19725 server_base.cc:1034] running on GCE node
I20260925 05:05:46.913723 19725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:46.915082 19725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:46.916321 19725 hybrid_clock.cc:648] HybridClock initialized: now 1790312746916267 us; error 46 us; skew 500 ppm
I20260925 05:05:46.916740 19725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:05:46 dist-test-slave-5n86 krb5kdc[19712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:46.919883 19725 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:05:46.921362 19725 webserver.cc:460] Webserver started at http://127.0.0.1:43497/ using document root <none> and password file <none>
I20260925 05:05:46.922016 19725 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:46.922165 19725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:46.922469 19725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:46.925374 19725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "03e348d181af4dda948f601a5d970bba"
format_stamp: "Formatted at 2026-09-25 05:05:46 on dist-test-slave-5n86"
server_key: "acf717fc1568a73530632a29a9c38899"
server_key_iv: "5b9d0afd788090e0f3e29e5d08fde278"
server_key_version: "encryptionkey@0"
I20260925 05:05:46.926123 19725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "03e348d181af4dda948f601a5d970bba"
format_stamp: "Formatted at 2026-09-25 05:05:46 on dist-test-slave-5n86"
server_key: "acf717fc1568a73530632a29a9c38899"
server_key_iv: "5b9d0afd788090e0f3e29e5d08fde278"
server_key_version: "encryptionkey@0"
I20260925 05:05:46.931020 19725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:05:46.934379 19740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:46.935662 19725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:05:46.935799 19725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "03e348d181af4dda948f601a5d970bba"
format_stamp: "Formatted at 2026-09-25 05:05:46 on dist-test-slave-5n86"
server_key: "acf717fc1568a73530632a29a9c38899"
server_key_iv: "5b9d0afd788090e0f3e29e5d08fde278"
server_key_version: "encryptionkey@0"
I20260925 05:05:46.935967 19725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:46.957396 19725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:46.961484 19725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:46.961705 19725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:46.979406 19725 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39209
I20260925 05:05:46.979432 19791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39209 every 8 connection(s)
I20260925 05:05:46.980795 19725 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:46.984248 19792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:46.985442 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 19725
I20260925 05:05:46.985633 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:05:46.986099 31708 external_mini_cluster.cc:1428] Setting key 86dd3dd63f428d1f1a49000383e9a2b3
I20260925 05:05:46.997260 19792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03e348d181af4dda948f601a5d970bba: Bootstrap starting.
I20260925 05:05:46.998805 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:46.987764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46527 (local address 127.0.0.1:39209)
0925 05:05:46.988605 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:46.988620 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:46.988660 (+ 40us) server_negotiation.cc:407] Connection header received
0925 05:05:46.990197 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:46.990220 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:46.990593 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:46.990904 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:46.991439 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:46.992760 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:46.994635 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:46.995286 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:46.995403 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:46.996698 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:46.996705 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:46.996718 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:46.997221 (+ 503us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:46.997293 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:46.997316 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:46.997486 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:46.997787 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 05:05:46.997939 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":375,"threads_started":1}
I20260925 05:05:47.000651 19792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03e348d181af4dda948f601a5d970bba: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:47.001969 19792 log.cc:824] T 00000000000000000000000000000000 P 03e348d181af4dda948f601a5d970bba: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:47.002743 31708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:05:47.004503 19792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03e348d181af4dda948f601a5d970bba: No bootstrap required, opened a new log
I20260925 05:05:47.013568 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:47.007132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33097 (local address 127.0.0.1:39209)
0925 05:05:47.007274 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:47.007278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:47.007594 (+ 316us) server_negotiation.cc:407] Connection header received
0925 05:05:47.007772 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:47.007775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:47.007829 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:47.007906 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:47.008423 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:47.009235 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:47.011079 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:47.011517 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:47.011620 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:47.012809 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:47.012814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:47.012817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:47.012832 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:47.012884 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:47.012899 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:47.012981 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:47.013181 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 05:05:47.013220 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 05:05:47.013877 19792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03e348d181af4dda948f601a5d970bba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e348d181af4dda948f601a5d970bba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39209 } }
I20260925 05:05:47.014242 19792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03e348d181af4dda948f601a5d970bba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:47.014364 19792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03e348d181af4dda948f601a5d970bba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e348d181af4dda948f601a5d970bba, State: Initialized, Role: FOLLOWER
I20260925 05:05:47.014897 19792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03e348d181af4dda948f601a5d970bba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e348d181af4dda948f601a5d970bba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39209 } }
I20260925 05:05:47.015136 19792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03e348d181af4dda948f601a5d970bba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:47.015278 19792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03e348d181af4dda948f601a5d970bba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:47.015451 19792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03e348d181af4dda948f601a5d970bba [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:47.017601 19792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03e348d181af4dda948f601a5d970bba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e348d181af4dda948f601a5d970bba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39209 } }
I20260925 05:05:47.018106 19792 leader_election.cc:302] T 00000000000000000000000000000000 P 03e348d181af4dda948f601a5d970bba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03e348d181af4dda948f601a5d970bba; no voters:
I20260925 05:05:47.018680 19792 leader_election.cc:288] T 00000000000000000000000000000000 P 03e348d181af4dda948f601a5d970bba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:47.018802 19802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03e348d181af4dda948f601a5d970bba [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:47.020313 19802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03e348d181af4dda948f601a5d970bba [term 1 LEADER]: Becoming Leader. State: Replica: 03e348d181af4dda948f601a5d970bba, State: Running, Role: LEADER
I20260925 05:05:47.020879 19802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03e348d181af4dda948f601a5d970bba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e348d181af4dda948f601a5d970bba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39209 } }
I20260925 05:05:47.021195 19792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03e348d181af4dda948f601a5d970bba [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:47.023835 19804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03e348d181af4dda948f601a5d970bba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03e348d181af4dda948f601a5d970bba. Latest consensus state: current_term: 1 leader_uuid: "03e348d181af4dda948f601a5d970bba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e348d181af4dda948f601a5d970bba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39209 } } }
I20260925 05:05:47.024289 19804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03e348d181af4dda948f601a5d970bba [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:47.025990 19810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:47.025996 19803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03e348d181af4dda948f601a5d970bba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03e348d181af4dda948f601a5d970bba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e348d181af4dda948f601a5d970bba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39209 } } }
I20260925 05:05:47.026361 19803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03e348d181af4dda948f601a5d970bba [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:47.031176 19810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:47.039912 19810 catalog_manager.cc:1347] Generated new cluster ID: 2e15a128838d44ebb9ac7892269b3528
I20260925 05:05:47.040012 19810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:05:47.059075 19810 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:47.060318 19810 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:47.069736 19810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03e348d181af4dda948f601a5d970bba: Generated new TSK 0
I20260925 05:05:47.070547 19810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:05:47.086789 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 19725
W20260925 05:05:47.088116 19787 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46527
W20260925 05:05:47.088884 31708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 19725: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:05:47.089082 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 19725
W20260925 05:05:47.105863 19710 connection.cc:535] client connection to 127.0.0.1:39209 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39209): Connection reset by peer (error 104)
2026-09-25T05:05:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1861 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:05:47 dist-test-slave-5n86 krb5kdc[19823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:05:47 dist-test-slave-5n86 krb5kdc[19823](info): set up 2 sockets
Sep 25 05:05:47 dist-test-slave-5n86 krb5kdc[19823](info): commencing operation
krb5kdc: starting...
W20260925 05:05:48.193392 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:05:48 dist-test-slave-5n86 krb5kdc[19823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:05:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:05:48.428716 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34457
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:48.742647 19836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:48.742940 19836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:48.743124 19836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:48.749904 19836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:48.750058 19836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:48.750166 19836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:48.750274 19836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:48.750360 19836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:48.759611 19836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34457
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:48.760905 19836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:48.762202 19836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:48.773988 19841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:48.773962 19842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:48.774014 19844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:48.774861 19836 server_base.cc:1034] running on GCE node
I20260925 05:05:48.775698 19836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:48.777237 19836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:48.778491 19836 hybrid_clock.cc:648] HybridClock initialized: now 1790312748778443 us; error 39 us; skew 500 ppm
I20260925 05:05:48.778880 19836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:05:48 dist-test-slave-5n86 krb5kdc[19823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:48.785593 19836 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:05:48.787194 19836 webserver.cc:460] Webserver started at http://127.0.0.1:35727/ using document root <none> and password file <none>
I20260925 05:05:48.787885 19836 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:48.788036 19836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:48.788345 19836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:48.791395 19836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "d649d315fa15444a9051efb62ed44537"
format_stamp: "Formatted at 2026-09-25 05:05:48 on dist-test-slave-5n86"
server_key: "41a55711bacb741966a6460c35528dd7"
server_key_iv: "e05744d038ac1d752034b54f7dbe9455"
server_key_version: "encryptionkey@0"
I20260925 05:05:48.792181 19836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "d649d315fa15444a9051efb62ed44537"
format_stamp: "Formatted at 2026-09-25 05:05:48 on dist-test-slave-5n86"
server_key: "41a55711bacb741966a6460c35528dd7"
server_key_iv: "e05744d038ac1d752034b54f7dbe9455"
server_key_version: "encryptionkey@0"
I20260925 05:05:48.797273 19836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 05:05:48.800722 19851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:48.801923 19836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:05:48.802110 19836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "d649d315fa15444a9051efb62ed44537"
format_stamp: "Formatted at 2026-09-25 05:05:48 on dist-test-slave-5n86"
server_key: "41a55711bacb741966a6460c35528dd7"
server_key_iv: "e05744d038ac1d752034b54f7dbe9455"
server_key_version: "encryptionkey@0"
I20260925 05:05:48.802302 19836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:48.807917 19836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:48.811744 19836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:48.812047 19836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:48.829721 19836 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34457
I20260925 05:05:48.829795 19902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34457 every 8 connection(s)
I20260925 05:05:48.831002 19836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:48.834522 19903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:48.839501 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 19836
I20260925 05:05:48.839684 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:05:48.840175 31708 external_mini_cluster.cc:1428] Setting key 6b8f7d3b90e15e334c8c6c261f78a7fd
I20260925 05:05:48.845213 19903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d649d315fa15444a9051efb62ed44537: Bootstrap starting.
I20260925 05:05:48.847391 19906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:48.841887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36535 (local address 127.0.0.1:34457)
0925 05:05:48.842651 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:48.842662 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:48.842704 (+ 42us) server_negotiation.cc:407] Connection header received
0925 05:05:48.844275 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:48.844297 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:48.844657 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:48.844956 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:48.845940 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:48.845946 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:48.845957 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:48.845970 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:48.846029 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:48.846050 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:48.846205 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:48.846340 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 05:05:48.846522 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":366,"threads_started":1}
I20260925 05:05:48.848766 19903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d649d315fa15444a9051efb62ed44537: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:48.849848 19903 log.cc:824] T 00000000000000000000000000000000 P d649d315fa15444a9051efb62ed44537: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:48.850873 31708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:05:48.852517 19903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d649d315fa15444a9051efb62ed44537: No bootstrap required, opened a new log
I20260925 05:05:48.857450 19906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:48.855052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43251 (local address 127.0.0.1:34457)
0925 05:05:48.855223 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:48.855229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:48.855640 (+ 411us) server_negotiation.cc:407] Connection header received
0925 05:05:48.855761 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:48.855766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:48.855824 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:48.855928 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:48.856810 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:48.856816 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:48.856821 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:48.856839 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:48.856885 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:48.856904 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:48.857013 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:48.857125 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 05:05:48.857176 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 05:05:48.860623 19903 raft_consensus.cc:348] T 00000000000000000000000000000000 P d649d315fa15444a9051efb62ed44537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d649d315fa15444a9051efb62ed44537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34457 } }
I20260925 05:05:48.860961 19903 raft_consensus.cc:374] T 00000000000000000000000000000000 P d649d315fa15444a9051efb62ed44537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:48.861066 19903 raft_consensus.cc:729] T 00000000000000000000000000000000 P d649d315fa15444a9051efb62ed44537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d649d315fa15444a9051efb62ed44537, State: Initialized, Role: FOLLOWER
I20260925 05:05:48.861739 19903 consensus_queue.cc:260] T 00000000000000000000000000000000 P d649d315fa15444a9051efb62ed44537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d649d315fa15444a9051efb62ed44537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34457 } }
I20260925 05:05:48.862021 19903 raft_consensus.cc:388] T 00000000000000000000000000000000 P d649d315fa15444a9051efb62ed44537 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:48.862176 19903 raft_consensus.cc:482] T 00000000000000000000000000000000 P d649d315fa15444a9051efb62ed44537 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:48.862353 19903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d649d315fa15444a9051efb62ed44537 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:48.864455 19903 raft_consensus.cc:504] T 00000000000000000000000000000000 P d649d315fa15444a9051efb62ed44537 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d649d315fa15444a9051efb62ed44537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34457 } }
I20260925 05:05:48.864954 19903 leader_election.cc:302] T 00000000000000000000000000000000 P d649d315fa15444a9051efb62ed44537 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d649d315fa15444a9051efb62ed44537; no voters:
I20260925 05:05:48.865517 19903 leader_election.cc:288] T 00000000000000000000000000000000 P d649d315fa15444a9051efb62ed44537 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:48.865657 19913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d649d315fa15444a9051efb62ed44537 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:48.866401 19913 raft_consensus.cc:686] T 00000000000000000000000000000000 P d649d315fa15444a9051efb62ed44537 [term 1 LEADER]: Becoming Leader. State: Replica: d649d315fa15444a9051efb62ed44537, State: Running, Role: LEADER
I20260925 05:05:48.866958 19913 consensus_queue.cc:237] T 00000000000000000000000000000000 P d649d315fa15444a9051efb62ed44537 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d649d315fa15444a9051efb62ed44537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34457 } }
I20260925 05:05:48.867623 19903 sys_catalog.cc:564] T 00000000000000000000000000000000 P d649d315fa15444a9051efb62ed44537 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:48.869926 19915 sys_catalog.cc:455] T 00000000000000000000000000000000 P d649d315fa15444a9051efb62ed44537 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d649d315fa15444a9051efb62ed44537. Latest consensus state: current_term: 1 leader_uuid: "d649d315fa15444a9051efb62ed44537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d649d315fa15444a9051efb62ed44537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34457 } } }
I20260925 05:05:48.869963 19914 sys_catalog.cc:455] T 00000000000000000000000000000000 P d649d315fa15444a9051efb62ed44537 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d649d315fa15444a9051efb62ed44537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d649d315fa15444a9051efb62ed44537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34457 } } }
I20260925 05:05:48.870321 19915 sys_catalog.cc:458] T 00000000000000000000000000000000 P d649d315fa15444a9051efb62ed44537 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:48.870373 19914 sys_catalog.cc:458] T 00000000000000000000000000000000 P d649d315fa15444a9051efb62ed44537 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:48.877153 19920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:48.881963 19920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:48.890228 19920 catalog_manager.cc:1347] Generated new cluster ID: 8cadfb1e9cf54d549cdc562b4291ee3c
I20260925 05:05:48.890331 19920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:05:48.908054 19920 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:48.908452 19920 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:48.917531 19920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d649d315fa15444a9051efb62ed44537: Generated new TSK 0
I20260925 05:05:48.918244 19920 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:05:48.927649 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 19836
W20260925 05:05:48.928699 19898 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36535
W20260925 05:05:48.929382 31708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 19836: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:05:48.929519 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 19836
2026-09-25T05:05:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1842 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:05:49 dist-test-slave-5n86 krb5kdc[19934](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:05:49 dist-test-slave-5n86 krb5kdc[19934](info): set up 2 sockets
Sep 25 05:05:49 dist-test-slave-5n86 krb5kdc[19934](info): commencing operation
krb5kdc: starting...
W20260925 05:05:50.037165 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:05:50 dist-test-slave-5n86 krb5kdc[19934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:05:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:05:50.275346 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39993
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:50.594101 19947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:50.594354 19947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:50.594525 19947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:50.601440 19947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:50.601531 19947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:50.601666 19947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:50.601773 19947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:50.601858 19947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:50.611030 19947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:50.612203 19947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:50.613456 19947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:50.624714 19955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:50.624684 19953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:50.625047 19952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:50.625486 19947 server_base.cc:1034] running on GCE node
I20260925 05:05:50.626230 19947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:50.627437 19947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:50.628649 19947 hybrid_clock.cc:648] HybridClock initialized: now 1790312750628595 us; error 54 us; skew 500 ppm
I20260925 05:05:50.628985 19947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:05:50 dist-test-slave-5n86 krb5kdc[19934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:50.632015 19947 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:05:50.633381 19947 webserver.cc:460] Webserver started at http://127.0.0.1:45227/ using document root <none> and password file <none>
I20260925 05:05:50.634019 19947 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:50.634172 19947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:50.634460 19947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:50.637293 19947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "692e20f5d81a4caa90e356c19338ffea"
format_stamp: "Formatted at 2026-09-25 05:05:50 on dist-test-slave-5n86"
server_key: "e0271873e2379df8fd9d9d2aa5d9d673"
server_key_iv: "06e3725ad7b8c66cdd56b41e15cdb9c0"
server_key_version: "encryptionkey@0"
I20260925 05:05:50.638029 19947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "692e20f5d81a4caa90e356c19338ffea"
format_stamp: "Formatted at 2026-09-25 05:05:50 on dist-test-slave-5n86"
server_key: "e0271873e2379df8fd9d9d2aa5d9d673"
server_key_iv: "06e3725ad7b8c66cdd56b41e15cdb9c0"
server_key_version: "encryptionkey@0"
I20260925 05:05:50.642822 19947 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 05:05:50.646052 19962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:50.647217 19947 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 05:05:50.647390 19947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "692e20f5d81a4caa90e356c19338ffea"
format_stamp: "Formatted at 2026-09-25 05:05:50 on dist-test-slave-5n86"
server_key: "e0271873e2379df8fd9d9d2aa5d9d673"
server_key_iv: "06e3725ad7b8c66cdd56b41e15cdb9c0"
server_key_version: "encryptionkey@0"
I20260925 05:05:50.647573 19947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:50.667451 19947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:50.671380 19947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:50.671659 19947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:50.689697 19947 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39993
I20260925 05:05:50.689735 20013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39993 every 8 connection(s)
I20260925 05:05:50.690968 19947 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:50.694636 20014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:50.698340 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 19947
I20260925 05:05:50.698518 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:05:50.699038 31708 external_mini_cluster.cc:1428] Setting key ca0d3259c81db7d2d7b7b7008ff3fc59
I20260925 05:05:50.706372 20014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 692e20f5d81a4caa90e356c19338ffea: Bootstrap starting.
I20260925 05:05:50.709247 20014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 692e20f5d81a4caa90e356c19338ffea: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:50.710204 20014 log.cc:824] T 00000000000000000000000000000000 P 692e20f5d81a4caa90e356c19338ffea: Log is configured to *not* fsync() on all Append() calls
Sep 25 05:05:50 dist-test-slave-5n86 krb5kdc[19934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 05:05:50.713341 20014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 692e20f5d81a4caa90e356c19338ffea: No bootstrap required, opened a new log
I20260925 05:05:50.720366 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:50.700848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58105 (local address 127.0.0.1:39993)
0925 05:05:50.701551 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:50.701562 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:50.701608 (+ 46us) server_negotiation.cc:407] Connection header received
0925 05:05:50.703161 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:50.703182 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:50.703554 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:50.703956 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:50.705471 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:50.706822 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:50.708018 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:50.708730 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:50.708920 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:50.713354 (+ 4434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:50.713364 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:50.713376 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:50.713451 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:50.716462 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:50.717296 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:50.717302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:50.717308 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:50.717464 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:50.717869 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:50.717873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:50.717875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:50.718377 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:50.718604 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:50.719033 (+ 429us) server_negotiation.cc:299] Negotiation successful
0925 05:05:50.719385 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":331,"threads_started":1}
I20260925 05:05:50.722080 20014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 692e20f5d81a4caa90e356c19338ffea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692e20f5d81a4caa90e356c19338ffea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } }
I20260925 05:05:50.722436 20014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 692e20f5d81a4caa90e356c19338ffea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:50.722560 20014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 692e20f5d81a4caa90e356c19338ffea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692e20f5d81a4caa90e356c19338ffea, State: Initialized, Role: FOLLOWER
I20260925 05:05:50.723148 20014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 692e20f5d81a4caa90e356c19338ffea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692e20f5d81a4caa90e356c19338ffea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } }
I20260925 05:05:50.723402 20014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 692e20f5d81a4caa90e356c19338ffea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:50.723546 20014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 692e20f5d81a4caa90e356c19338ffea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:50.723721 20014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 692e20f5d81a4caa90e356c19338ffea [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:50.723938 31708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:05:50.725939 20014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 692e20f5d81a4caa90e356c19338ffea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692e20f5d81a4caa90e356c19338ffea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } }
I20260925 05:05:50.726403 20014 leader_election.cc:302] T 00000000000000000000000000000000 P 692e20f5d81a4caa90e356c19338ffea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 692e20f5d81a4caa90e356c19338ffea; no voters:
I20260925 05:05:50.727061 20014 leader_election.cc:288] T 00000000000000000000000000000000 P 692e20f5d81a4caa90e356c19338ffea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:50.727144 20021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 692e20f5d81a4caa90e356c19338ffea [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:50.729463 20014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 692e20f5d81a4caa90e356c19338ffea [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:50.734031 20021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 692e20f5d81a4caa90e356c19338ffea [term 1 LEADER]: Becoming Leader. State: Replica: 692e20f5d81a4caa90e356c19338ffea, State: Running, Role: LEADER
I20260925 05:05:50.734740 20021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 692e20f5d81a4caa90e356c19338ffea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692e20f5d81a4caa90e356c19338ffea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } }
I20260925 05:05:50.740447 20023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 692e20f5d81a4caa90e356c19338ffea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 692e20f5d81a4caa90e356c19338ffea. Latest consensus state: current_term: 1 leader_uuid: "692e20f5d81a4caa90e356c19338ffea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692e20f5d81a4caa90e356c19338ffea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } } }
I20260925 05:05:50.741266 20023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 692e20f5d81a4caa90e356c19338ffea [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:50.740809 20022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 692e20f5d81a4caa90e356c19338ffea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "692e20f5d81a4caa90e356c19338ffea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692e20f5d81a4caa90e356c19338ffea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } } }
I20260925 05:05:50.742050 20022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 692e20f5d81a4caa90e356c19338ffea [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:50.749874 20036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:50.755312 20036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:50.761005 20042 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 05:05:50.764684 20036 catalog_manager.cc:1347] Generated new cluster ID: d1a079f148a2440c85efe4f97b6cd641
I20260925 05:05:50.764812 20036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:05:50.767059 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:50.737278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33545 (local address 127.0.0.1:39993)
0925 05:05:50.737460 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:50.737465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:50.749930 (+ 12465us) server_negotiation.cc:407] Connection header received
0925 05:05:50.750127 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:50.750136 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:50.750247 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:50.750384 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:50.752621 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:50.753788 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:50.755658 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:50.756613 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:50.756744 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:50.760380 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:50.760387 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:50.760393 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:50.760430 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:50.764112 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:50.764988 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:50.764993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:50.765015 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:50.765129 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:50.765522 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:50.765527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:50.765530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:50.765856 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:50.766133 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:50.766444 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 05:05:50.766654 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 05:05:50.773713 20036 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:50.774885 20036 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:50.788239 20036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 692e20f5d81a4caa90e356c19338ffea: Generated new TSK 0
I20260925 05:05:50.789062 20036 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:05:50.796829 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 19947
I20260925 05:05:50.797768 20009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:50.885987 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 19947
2026-09-25T05:05:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1954 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:05:50 dist-test-slave-5n86 krb5kdc[20047](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:05:50 dist-test-slave-5n86 krb5kdc[20047](info): set up 2 sockets
Sep 25 05:05:50 dist-test-slave-5n86 krb5kdc[20047](info): commencing operation
krb5kdc: starting...
W20260925 05:05:51.981156 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:05:52 dist-test-slave-5n86 krb5kdc[20047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:05:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:05:52.222111 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38037
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:52.546967 20060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:52.547254 20060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:52.547345 20060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:52.554481 20060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:52.554589 20060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:52.554634 20060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:52.554694 20060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:52.554737 20060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:52.564246 20060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:52.565502 20060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:52.566731 20060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:52.578347 20066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:52.578416 20065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:52.578423 20068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:52.579749 20060 server_base.cc:1034] running on GCE node
I20260925 05:05:52.580520 20060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:52.581976 20060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:52.583207 20060 hybrid_clock.cc:648] HybridClock initialized: now 1790312752583162 us; error 39 us; skew 500 ppm
I20260925 05:05:52.583554 20060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:05:52 dist-test-slave-5n86 krb5kdc[20047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:52.586709 20060 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:05:52.588191 20060 webserver.cc:460] Webserver started at http://127.0.0.1:45321/ using document root <none> and password file <none>
I20260925 05:05:52.588977 20060 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:52.589133 20060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:52.589437 20060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:52.592427 20060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "1c16235638294db0a6434d36f13487cd"
format_stamp: "Formatted at 2026-09-25 05:05:52 on dist-test-slave-5n86"
server_key: "a840d7df5e06ee7363143d6304cd2d75"
server_key_iv: "9ff7cb75859164334053acbc1a2052bb"
server_key_version: "encryptionkey@0"
I20260925 05:05:52.593252 20060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "1c16235638294db0a6434d36f13487cd"
format_stamp: "Formatted at 2026-09-25 05:05:52 on dist-test-slave-5n86"
server_key: "a840d7df5e06ee7363143d6304cd2d75"
server_key_iv: "9ff7cb75859164334053acbc1a2052bb"
server_key_version: "encryptionkey@0"
I20260925 05:05:52.598094 20060 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 05:05:52.601423 20075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:52.602807 20060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:05:52.602996 20060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "1c16235638294db0a6434d36f13487cd"
format_stamp: "Formatted at 2026-09-25 05:05:52 on dist-test-slave-5n86"
server_key: "a840d7df5e06ee7363143d6304cd2d75"
server_key_iv: "9ff7cb75859164334053acbc1a2052bb"
server_key_version: "encryptionkey@0"
I20260925 05:05:52.603204 20060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:52.619330 20060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:52.623468 20060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:52.623747 20060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:52.642326 20060 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38037
I20260925 05:05:52.642369 20126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38037 every 8 connection(s)
I20260925 05:05:52.643636 20060 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:52.644016 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 20060
I20260925 05:05:52.644199 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:05:52.644829 31708 external_mini_cluster.cc:1428] Setting key 826afdf5742cc459493e17492ee7075f
I20260925 05:05:52.647943 20127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:52.657439 20130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:52.646590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41417 (local address 127.0.0.1:38037)
0925 05:05:52.647330 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:52.647342 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:52.647381 (+ 39us) server_negotiation.cc:407] Connection header received
0925 05:05:52.649114 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:52.649136 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:52.649499 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:52.649880 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:52.650421 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:52.651697 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:52.653072 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:52.653739 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:52.653937 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:52.655162 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:52.655170 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:52.655186 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:52.655744 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:52.655814 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:52.655838 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:52.655975 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:52.656209 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 05:05:52.656452 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":336,"threads_started":1}
I20260925 05:05:52.658876 20127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c16235638294db0a6434d36f13487cd: Bootstrap starting.
I20260925 05:05:52.661151 31708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:05:52.662163 20127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c16235638294db0a6434d36f13487cd: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:52.663123 20127 log.cc:824] T 00000000000000000000000000000000 P 1c16235638294db0a6434d36f13487cd: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:52.666453 20127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c16235638294db0a6434d36f13487cd: No bootstrap required, opened a new log
I20260925 05:05:52.671558 20130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:52.665667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53527 (local address 127.0.0.1:38037)
0925 05:05:52.665848 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:52.665857 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:52.666188 (+ 331us) server_negotiation.cc:407] Connection header received
0925 05:05:52.666409 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:52.666413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:52.666471 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:52.666572 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:52.667083 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:52.667799 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:52.669162 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:52.669637 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:52.669780 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:52.670841 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:52.670846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:52.670849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:52.670865 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:52.670903 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:52.670920 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:52.671026 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:52.671175 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 05:05:52.671273 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 05:05:52.674134 20127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c16235638294db0a6434d36f13487cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c16235638294db0a6434d36f13487cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38037 } }
I20260925 05:05:52.674489 20127 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c16235638294db0a6434d36f13487cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:52.674598 20127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c16235638294db0a6434d36f13487cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c16235638294db0a6434d36f13487cd, State: Initialized, Role: FOLLOWER
I20260925 05:05:52.675103 20127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c16235638294db0a6434d36f13487cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c16235638294db0a6434d36f13487cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38037 } }
I20260925 05:05:52.675338 20127 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c16235638294db0a6434d36f13487cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:52.675478 20127 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c16235638294db0a6434d36f13487cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:52.675652 20127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c16235638294db0a6434d36f13487cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:52.678056 20127 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c16235638294db0a6434d36f13487cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c16235638294db0a6434d36f13487cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38037 } }
I20260925 05:05:52.678566 20127 leader_election.cc:302] T 00000000000000000000000000000000 P 1c16235638294db0a6434d36f13487cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c16235638294db0a6434d36f13487cd; no voters:
I20260925 05:05:52.679131 20127 leader_election.cc:288] T 00000000000000000000000000000000 P 1c16235638294db0a6434d36f13487cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:52.679216 20137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c16235638294db0a6434d36f13487cd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:52.679919 20137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c16235638294db0a6434d36f13487cd [term 1 LEADER]: Becoming Leader. State: Replica: 1c16235638294db0a6434d36f13487cd, State: Running, Role: LEADER
I20260925 05:05:52.680482 20137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c16235638294db0a6434d36f13487cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c16235638294db0a6434d36f13487cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38037 } }
I20260925 05:05:52.681118 20127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c16235638294db0a6434d36f13487cd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:52.683357 20138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c16235638294db0a6434d36f13487cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c16235638294db0a6434d36f13487cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c16235638294db0a6434d36f13487cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38037 } } }
I20260925 05:05:52.683748 20138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c16235638294db0a6434d36f13487cd [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:52.683897 20139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c16235638294db0a6434d36f13487cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c16235638294db0a6434d36f13487cd. Latest consensus state: current_term: 1 leader_uuid: "1c16235638294db0a6434d36f13487cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c16235638294db0a6434d36f13487cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38037 } } }
I20260925 05:05:52.684212 20139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c16235638294db0a6434d36f13487cd [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:52.685102 20146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:52.691000 20146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:52.701102 20146 catalog_manager.cc:1347] Generated new cluster ID: 27c69ec73ec44c35b4ce9d6c6830e119
I20260925 05:05:52.701200 20146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:05:52.719972 20146 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:52.721771 20146 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:52.731712 20146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c16235638294db0a6434d36f13487cd: Generated new TSK 0
I20260925 05:05:52.732573 20146 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:05:52.743667 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 20060
W20260925 05:05:52.744737 20122 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41417
W20260925 05:05:52.745407 31708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 20060: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:05:52.745551 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 20060
2026-09-25T05:05:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1856 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:05:52 dist-test-slave-5n86 krb5kdc[20158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:05:52 dist-test-slave-5n86 krb5kdc[20158](info): set up 2 sockets
Sep 25 05:05:52 dist-test-slave-5n86 krb5kdc[20158](info): commencing operation
krb5kdc: starting...
W20260925 05:05:53.849077 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:05:54 dist-test-slave-5n86 krb5kdc[20158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:05:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:05:54.097656 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34539
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:54.415082 20171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:54.415342 20171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:54.415491 20171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:54.422482 20171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:54.422583 20171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:54.422673 20171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:54.422776 20171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:54.422865 20171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:54.432343 20171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:54.433593 20171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:54.434857 20171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:54.446386 20179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:54.446445 20177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:54.446470 20176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:54.447206 20171 server_base.cc:1034] running on GCE node
I20260925 05:05:54.447956 20171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:54.449337 20171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:54.450536 20171 hybrid_clock.cc:648] HybridClock initialized: now 1790312754450499 us; error 30 us; skew 500 ppm
I20260925 05:05:54.450868 20171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:05:54 dist-test-slave-5n86 krb5kdc[20158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:54.462386 20171 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:05:54.463879 20171 webserver.cc:460] Webserver started at http://127.0.0.1:46757/ using document root <none> and password file <none>
I20260925 05:05:54.464605 20171 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:54.464758 20171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:54.465059 20171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:54.467939 20171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "80354fb5c1334ad6b4fcee813652b743"
format_stamp: "Formatted at 2026-09-25 05:05:54 on dist-test-slave-5n86"
server_key: "03a761ef3ae11aee12f9abadec6aa25d"
server_key_iv: "1e26f36c514ba5354e66e7e4bef8d5da"
server_key_version: "encryptionkey@0"
I20260925 05:05:54.468761 20171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "80354fb5c1334ad6b4fcee813652b743"
format_stamp: "Formatted at 2026-09-25 05:05:54 on dist-test-slave-5n86"
server_key: "03a761ef3ae11aee12f9abadec6aa25d"
server_key_iv: "1e26f36c514ba5354e66e7e4bef8d5da"
server_key_version: "encryptionkey@0"
I20260925 05:05:54.473584 20171 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 05:05:54.476790 20186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:54.477994 20171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:05:54.478173 20171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "80354fb5c1334ad6b4fcee813652b743"
format_stamp: "Formatted at 2026-09-25 05:05:54 on dist-test-slave-5n86"
server_key: "03a761ef3ae11aee12f9abadec6aa25d"
server_key_iv: "1e26f36c514ba5354e66e7e4bef8d5da"
server_key_version: "encryptionkey@0"
I20260925 05:05:54.478360 20171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:54.483829 20171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:54.487478 20171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:54.487766 20171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:54.505558 20171 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34539
I20260925 05:05:54.505633 20237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34539 every 8 connection(s)
I20260925 05:05:54.506821 20171 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:54.509461 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 20171
I20260925 05:05:54.509644 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:05:54.510149 31708 external_mini_cluster.cc:1428] Setting key 298d4bc510cb30c438d38187c6408877
I20260925 05:05:54.510758 20238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:54.517424 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:54.511842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57551 (local address 127.0.0.1:34539)
0925 05:05:54.512638 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:54.512649 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:54.512689 (+ 40us) server_negotiation.cc:407] Connection header received
0925 05:05:54.514114 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:54.514133 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:54.514463 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:54.514843 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:54.515841 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:54.515847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:54.515858 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:54.515871 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:54.515936 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:54.515955 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:54.516210 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:54.516350 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 05:05:54.516518 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":340,"threads_started":1}
I20260925 05:05:54.520882 31708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:05:54.523459 20238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80354fb5c1334ad6b4fcee813652b743: Bootstrap starting.
I20260925 05:05:54.526883 20238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80354fb5c1334ad6b4fcee813652b743: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:54.527253 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:54.525032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35261 (local address 127.0.0.1:34539)
0925 05:05:54.525203 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:54.525210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:54.525522 (+ 312us) server_negotiation.cc:407] Connection header received
0925 05:05:54.525635 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:54.525642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:54.525703 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:54.525808 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:54.526627 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:54.526635 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:54.526643 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:54.526663 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:54.526713 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:54.526743 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:54.526872 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:54.526922 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 05:05:54.526974 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 05:05:54.528023 20238 log.cc:824] T 00000000000000000000000000000000 P 80354fb5c1334ad6b4fcee813652b743: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:54.530496 20238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80354fb5c1334ad6b4fcee813652b743: No bootstrap required, opened a new log
I20260925 05:05:54.537276 20238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80354fb5c1334ad6b4fcee813652b743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80354fb5c1334ad6b4fcee813652b743" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34539 } }
I20260925 05:05:54.537603 20238 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80354fb5c1334ad6b4fcee813652b743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:54.537711 20238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80354fb5c1334ad6b4fcee813652b743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80354fb5c1334ad6b4fcee813652b743, State: Initialized, Role: FOLLOWER
I20260925 05:05:54.538297 20238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80354fb5c1334ad6b4fcee813652b743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80354fb5c1334ad6b4fcee813652b743" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34539 } }
I20260925 05:05:54.538517 20238 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80354fb5c1334ad6b4fcee813652b743 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:54.538661 20238 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80354fb5c1334ad6b4fcee813652b743 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:54.538827 20238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80354fb5c1334ad6b4fcee813652b743 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:54.541015 20238 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80354fb5c1334ad6b4fcee813652b743 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80354fb5c1334ad6b4fcee813652b743" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34539 } }
I20260925 05:05:54.541469 20238 leader_election.cc:302] T 00000000000000000000000000000000 P 80354fb5c1334ad6b4fcee813652b743 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80354fb5c1334ad6b4fcee813652b743; no voters:
I20260925 05:05:54.542021 20238 leader_election.cc:288] T 00000000000000000000000000000000 P 80354fb5c1334ad6b4fcee813652b743 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:54.542161 20248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80354fb5c1334ad6b4fcee813652b743 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:54.542886 20248 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80354fb5c1334ad6b4fcee813652b743 [term 1 LEADER]: Becoming Leader. State: Replica: 80354fb5c1334ad6b4fcee813652b743, State: Running, Role: LEADER
I20260925 05:05:54.543375 20248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80354fb5c1334ad6b4fcee813652b743 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80354fb5c1334ad6b4fcee813652b743" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34539 } }
I20260925 05:05:54.543944 20238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80354fb5c1334ad6b4fcee813652b743 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:54.550128 20249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80354fb5c1334ad6b4fcee813652b743 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80354fb5c1334ad6b4fcee813652b743" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80354fb5c1334ad6b4fcee813652b743" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34539 } } }
I20260925 05:05:54.551116 20249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80354fb5c1334ad6b4fcee813652b743 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:54.550609 20250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80354fb5c1334ad6b4fcee813652b743 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80354fb5c1334ad6b4fcee813652b743. Latest consensus state: current_term: 1 leader_uuid: "80354fb5c1334ad6b4fcee813652b743" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80354fb5c1334ad6b4fcee813652b743" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34539 } } }
I20260925 05:05:54.551436 20250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80354fb5c1334ad6b4fcee813652b743 [sys.catalog]: This master's current role is: LEADER
W20260925 05:05:54.555725 20262 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 80354fb5c1334ad6b4fcee813652b743: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:05:54.555852 20262 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:05:54.555985 20263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:54.557649 20263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:54.565491 20263 catalog_manager.cc:1347] Generated new cluster ID: 8f52c7a6cfbd4749b5602a419f40975e
I20260925 05:05:54.565604 20263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:05:54.577724 20263 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:54.578114 20263 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:54.585381 20263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80354fb5c1334ad6b4fcee813652b743: Generated new TSK 0
I20260925 05:05:54.586107 20263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:05:54.596844 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 20171
W20260925 05:05:54.597884 20233 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57551
W20260925 05:05:54.598531 31708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 20171: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:05:54.598670 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 20171
2026-09-25T05:05:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1855 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:05:54 dist-test-slave-5n86 krb5kdc[20269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:05:54 dist-test-slave-5n86 krb5kdc[20269](info): set up 2 sockets
Sep 25 05:05:54 dist-test-slave-5n86 krb5kdc[20269](info): commencing operation
krb5kdc: starting...
W20260925 05:05:55.693248 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:05:55 dist-test-slave-5n86 krb5kdc[20269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:05:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:05:55.931556 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44377
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:56.253273 20282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:56.253590 20282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:56.253760 20282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:56.260795 20282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:56.260934 20282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:56.261029 20282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:56.261142 20282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:56.261234 20282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:56.270596 20282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:56.271814 20282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:56.273123 20282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:56.284677 20288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:56.284758 20287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:56.284715 20290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:56.285527 20282 server_base.cc:1034] running on GCE node
I20260925 05:05:56.286238 20282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:56.287524 20282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:56.288728 20282 hybrid_clock.cc:648] HybridClock initialized: now 1790312756288698 us; error 31 us; skew 500 ppm
I20260925 05:05:56.289060 20282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:05:56 dist-test-slave-5n86 krb5kdc[20269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:56.292017 20282 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:05:56.293460 20282 webserver.cc:460] Webserver started at http://127.0.0.1:43943/ using document root <none> and password file <none>
I20260925 05:05:56.294080 20282 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:56.294207 20282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:56.294507 20282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:56.297426 20282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "4453f324181145569f69995309ed4c7d"
format_stamp: "Formatted at 2026-09-25 05:05:56 on dist-test-slave-5n86"
server_key: "b51f1377b385227cc1b14ac4101d0140"
server_key_iv: "51d5eecd19a43f611a1348d41702c5c1"
server_key_version: "encryptionkey@0"
I20260925 05:05:56.298179 20282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "4453f324181145569f69995309ed4c7d"
format_stamp: "Formatted at 2026-09-25 05:05:56 on dist-test-slave-5n86"
server_key: "b51f1377b385227cc1b14ac4101d0140"
server_key_iv: "51d5eecd19a43f611a1348d41702c5c1"
server_key_version: "encryptionkey@0"
I20260925 05:05:56.303073 20282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 05:05:56.306363 20297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:56.307641 20282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:05:56.307811 20282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "4453f324181145569f69995309ed4c7d"
format_stamp: "Formatted at 2026-09-25 05:05:56 on dist-test-slave-5n86"
server_key: "b51f1377b385227cc1b14ac4101d0140"
server_key_iv: "51d5eecd19a43f611a1348d41702c5c1"
server_key_version: "encryptionkey@0"
I20260925 05:05:56.307979 20282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:56.329898 20282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:56.333844 20282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:56.334131 20282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:56.352273 20282 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44377
I20260925 05:05:56.352304 20348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44377 every 8 connection(s)
I20260925 05:05:56.353622 20282 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:56.357357 20349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:56.363075 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 20282
I20260925 05:05:56.363255 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:05:56.363775 31708 external_mini_cluster.cc:1428] Setting key 9f35395d99af0856eb9b60ee3a372b6a
I20260925 05:05:56.368261 20349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4453f324181145569f69995309ed4c7d: Bootstrap starting.
I20260925 05:05:56.371850 20349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4453f324181145569f69995309ed4c7d: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:56.373080 20349 log.cc:824] T 00000000000000000000000000000000 P 4453f324181145569f69995309ed4c7d: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:56.376114 20349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4453f324181145569f69995309ed4c7d: No bootstrap required, opened a new log
Sep 25 05:05:56 dist-test-slave-5n86 krb5kdc[20269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 05:05:56.386831 20349 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4453f324181145569f69995309ed4c7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4453f324181145569f69995309ed4c7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44377 } }
I20260925 05:05:56.387281 20349 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4453f324181145569f69995309ed4c7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:56.387388 20349 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4453f324181145569f69995309ed4c7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4453f324181145569f69995309ed4c7d, State: Initialized, Role: FOLLOWER
I20260925 05:05:56.388019 20349 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4453f324181145569f69995309ed4c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4453f324181145569f69995309ed4c7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44377 } }
I20260925 05:05:56.388302 20349 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4453f324181145569f69995309ed4c7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:56.388445 20349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4453f324181145569f69995309ed4c7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:56.388706 20352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:56.365750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:44813 (local address 127.0.0.1:44377)
0925 05:05:56.366574 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:56.366586 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:56.366627 (+ 41us) server_negotiation.cc:407] Connection header received
0925 05:05:56.368272 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:56.368294 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:56.368809 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:56.369116 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:56.370897 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:56.372265 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:56.374012 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:56.374782 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:56.374905 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:56.379681 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:56.379691 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:56.379710 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:56.379756 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:56.384028 (+ 4272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:56.384992 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:56.385001 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:56.385009 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:56.385225 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:56.385625 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:56.385629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:56.385633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:56.386360 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:56.386675 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:56.387023 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 05:05:56.387358 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":386,"threads_started":1}
I20260925 05:05:56.388703 20349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4453f324181145569f69995309ed4c7d [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:56.391078 20349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4453f324181145569f69995309ed4c7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4453f324181145569f69995309ed4c7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44377 } }
I20260925 05:05:56.391598 20349 leader_election.cc:302] T 00000000000000000000000000000000 P 4453f324181145569f69995309ed4c7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4453f324181145569f69995309ed4c7d; no voters:
I20260925 05:05:56.392182 20349 leader_election.cc:288] T 00000000000000000000000000000000 P 4453f324181145569f69995309ed4c7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:56.392267 20354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4453f324181145569f69995309ed4c7d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:56.392637 31708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:05:56.393031 20354 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4453f324181145569f69995309ed4c7d [term 1 LEADER]: Becoming Leader. State: Replica: 4453f324181145569f69995309ed4c7d, State: Running, Role: LEADER
I20260925 05:05:56.393599 20354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4453f324181145569f69995309ed4c7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4453f324181145569f69995309ed4c7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44377 } }
I20260925 05:05:56.395995 20356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4453f324181145569f69995309ed4c7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4453f324181145569f69995309ed4c7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4453f324181145569f69995309ed4c7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44377 } } }
I20260925 05:05:56.396342 20356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4453f324181145569f69995309ed4c7d [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:56.401357 20354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4453f324181145569f69995309ed4c7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4453f324181145569f69995309ed4c7d. Latest consensus state: current_term: 1 leader_uuid: "4453f324181145569f69995309ed4c7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4453f324181145569f69995309ed4c7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44377 } } }
I20260925 05:05:56.401996 20354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4453f324181145569f69995309ed4c7d [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:56.407253 20349 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4453f324181145569f69995309ed4c7d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:56.407783 20358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:56.412904 20358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:56.429243 20352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:56.407339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:35949 (local address 127.0.0.1:44377)
0925 05:05:56.407545 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:56.407553 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:56.407959 (+ 406us) server_negotiation.cc:407] Connection header received
0925 05:05:56.408210 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:56.408215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:56.408296 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:56.408385 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:56.410855 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:56.412072 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:56.414486 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:56.415422 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:56.415540 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:56.419753 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:56.419760 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:56.419766 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:56.419813 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:56.424055 (+ 4242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:56.425349 (+ 1294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:56.425358 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:56.425364 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:56.425536 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:56.426224 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:56.426231 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:56.426238 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:56.426758 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:56.427078 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:56.428422 (+ 1344us) server_negotiation.cc:299] Negotiation successful
0925 05:05:56.428673 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 05:05:56.440804 20358 catalog_manager.cc:1347] Generated new cluster ID: d037f73778f24cedb1b464d310c9c2c7
I20260925 05:05:56.440914 20358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:05:56.484736 20358 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:56.486557 20358 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:56.500552 20358 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4453f324181145569f69995309ed4c7d: Generated new TSK 0
I20260925 05:05:56.501405 20358 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:05:56.516516 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 20282
I20260925 05:05:56.517549 20344 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:05:56.612349 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 20282
W20260925 05:05:56.629570 20267 connection.cc:535] client connection to 127.0.0.1:44377 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44377): Connection reset by peer (error 104)
2026-09-25T05:05:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2014 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:05:56 dist-test-slave-5n86 krb5kdc[20382](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:05:56 dist-test-slave-5n86 krb5kdc[20382](info): set up 2 sockets
Sep 25 05:05:56 dist-test-slave-5n86 krb5kdc[20382](info): commencing operation
krb5kdc: starting...
W20260925 05:05:57.717118 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:05:57 dist-test-slave-5n86 krb5kdc[20382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:05:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:05:57.960727 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35597
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:58.282317 20395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:58.282588 20395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:58.282677 20395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:58.289685 20395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:58.289786 20395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:58.289839 20395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:58.289898 20395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:58.289942 20395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:58.299651 20395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:05:58.300911 20395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:58.302309 20395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:58.314426 20403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:58.314515 20401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:58.314384 20400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:58.315471 20395 server_base.cc:1034] running on GCE node
I20260925 05:05:58.316321 20395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:58.318003 20395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:58.319275 20395 hybrid_clock.cc:648] HybridClock initialized: now 1790312758319217 us; error 48 us; skew 500 ppm
I20260925 05:05:58.319641 20395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:05:58 dist-test-slave-5n86 krb5kdc[20382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:58.322944 20395 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:05:58.324882 20395 webserver.cc:460] Webserver started at http://127.0.0.1:45665/ using document root <none> and password file <none>
I20260925 05:05:58.325636 20395 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:58.325783 20395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:58.326112 20395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:58.329175 20395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "9234475bd89d44999d97e7c09893d1bc"
format_stamp: "Formatted at 2026-09-25 05:05:58 on dist-test-slave-5n86"
server_key: "b784bf8730aa40b3b162c8d4888ec17c"
server_key_iv: "9c83f258c6a0ff4febc660705cf41d39"
server_key_version: "encryptionkey@0"
I20260925 05:05:58.329958 20395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "9234475bd89d44999d97e7c09893d1bc"
format_stamp: "Formatted at 2026-09-25 05:05:58 on dist-test-slave-5n86"
server_key: "b784bf8730aa40b3b162c8d4888ec17c"
server_key_iv: "9c83f258c6a0ff4febc660705cf41d39"
server_key_version: "encryptionkey@0"
I20260925 05:05:58.335001 20395 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:05:58.338414 20410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:58.339660 20395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260925 05:05:58.339843 20395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "9234475bd89d44999d97e7c09893d1bc"
format_stamp: "Formatted at 2026-09-25 05:05:58 on dist-test-slave-5n86"
server_key: "b784bf8730aa40b3b162c8d4888ec17c"
server_key_iv: "9c83f258c6a0ff4febc660705cf41d39"
server_key_version: "encryptionkey@0"
I20260925 05:05:58.340030 20395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:58.382294 20395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:58.386915 20395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:58.387220 20395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:58.405474 20395 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35597
I20260925 05:05:58.405511 20461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35597 every 8 connection(s)
I20260925 05:05:58.406800 20395 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:58.410409 20462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:58.414760 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 20395
I20260925 05:05:58.414927 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:05:58.415385 31708 external_mini_cluster.cc:1428] Setting key 9dae95ad1a806a999b48e2fea2a4eb56
I20260925 05:05:58.422407 20462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9234475bd89d44999d97e7c09893d1bc: Bootstrap starting.
I20260925 05:05:58.425909 20462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9234475bd89d44999d97e7c09893d1bc: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:58.427160 20462 log.cc:824] T 00000000000000000000000000000000 P 9234475bd89d44999d97e7c09893d1bc: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:58.430233 20462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9234475bd89d44999d97e7c09893d1bc: No bootstrap required, opened a new log
I20260925 05:05:58.430466 20465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:58.417133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:40225 (local address 127.0.0.1:35597)
0925 05:05:58.417885 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:58.417904 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:58.417952 (+ 48us) server_negotiation.cc:407] Connection header received
0925 05:05:58.420611 (+ 2659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:58.420640 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:58.421107 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:58.421506 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:58.422008 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:58.423986 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:58.425410 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:58.426494 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:58.426661 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:58.427957 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:58.427972 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:58.427994 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:58.428742 (+ 748us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:58.428813 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:58.428841 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:58.428956 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:58.429350 (+ 394us) server_negotiation.cc:299] Negotiation successful
0925 05:05:58.429552 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":289,"threads_started":1}
I20260925 05:05:58.434401 31708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:05:58.438499 20462 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9234475bd89d44999d97e7c09893d1bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9234475bd89d44999d97e7c09893d1bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35597 } }
I20260925 05:05:58.439028 20462 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9234475bd89d44999d97e7c09893d1bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:58.439136 20462 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9234475bd89d44999d97e7c09893d1bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9234475bd89d44999d97e7c09893d1bc, State: Initialized, Role: FOLLOWER
I20260925 05:05:58.439745 20462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9234475bd89d44999d97e7c09893d1bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9234475bd89d44999d97e7c09893d1bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35597 } }
I20260925 05:05:58.439977 20462 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9234475bd89d44999d97e7c09893d1bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:58.440119 20462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9234475bd89d44999d97e7c09893d1bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:58.440325 20462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9234475bd89d44999d97e7c09893d1bc [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:58.442586 20462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9234475bd89d44999d97e7c09893d1bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9234475bd89d44999d97e7c09893d1bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35597 } }
I20260925 05:05:58.443081 20462 leader_election.cc:302] T 00000000000000000000000000000000 P 9234475bd89d44999d97e7c09893d1bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9234475bd89d44999d97e7c09893d1bc; no voters:
I20260925 05:05:58.443840 20462 leader_election.cc:288] T 00000000000000000000000000000000 P 9234475bd89d44999d97e7c09893d1bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:58.443936 20472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9234475bd89d44999d97e7c09893d1bc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:58.445382 20465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:58.438682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:53901 (local address 127.0.0.1:35597)
0925 05:05:58.438866 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:58.438873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:58.439324 (+ 451us) server_negotiation.cc:407] Connection header received
0925 05:05:58.439547 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:58.439557 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:58.439615 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:58.439720 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:58.440250 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:58.441127 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:58.442458 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:58.442998 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:58.443137 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:58.444502 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:58.444511 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:58.444517 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:58.444562 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:58.444605 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:58.444626 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:58.444726 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:58.445019 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 05:05:58.445071 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 05:05:58.446334 20462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9234475bd89d44999d97e7c09893d1bc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:58.448817 20472 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9234475bd89d44999d97e7c09893d1bc [term 1 LEADER]: Becoming Leader. State: Replica: 9234475bd89d44999d97e7c09893d1bc, State: Running, Role: LEADER
I20260925 05:05:58.449502 20472 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9234475bd89d44999d97e7c09893d1bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9234475bd89d44999d97e7c09893d1bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35597 } }
I20260925 05:05:58.453203 20473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9234475bd89d44999d97e7c09893d1bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9234475bd89d44999d97e7c09893d1bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9234475bd89d44999d97e7c09893d1bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35597 } } }
I20260925 05:05:58.453598 20473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9234475bd89d44999d97e7c09893d1bc [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:58.454186 20474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9234475bd89d44999d97e7c09893d1bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9234475bd89d44999d97e7c09893d1bc. Latest consensus state: current_term: 1 leader_uuid: "9234475bd89d44999d97e7c09893d1bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9234475bd89d44999d97e7c09893d1bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35597 } } }
I20260925 05:05:58.454681 20474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9234475bd89d44999d97e7c09893d1bc [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:58.457957 20482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:58.463147 20482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:58.471678 20482 catalog_manager.cc:1347] Generated new cluster ID: 505471976f614d5aa8a2da079353e1e1
I20260925 05:05:58.472193 20482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:05:58.493956 20482 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:58.495229 20482 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:58.511059 20482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9234475bd89d44999d97e7c09893d1bc: Generated new TSK 0
I20260925 05:05:58.512054 20482 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:05:58.521689 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 20395
W20260925 05:05:58.522794 20457 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.152:40225
W20260925 05:05:58.523514 31708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 20395: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:05:58.523662 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 20395
W20260925 05:05:58.539872 20380 connection.cc:535] client connection to 127.0.0.1:35597 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35597): Connection reset by peer (error 104)
2026-09-25T05:05:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1910 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:05:58 dist-test-slave-5n86 krb5kdc[20493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:05:58 dist-test-slave-5n86 krb5kdc[20493](info): set up 2 sockets
Sep 25 05:05:58 dist-test-slave-5n86 krb5kdc[20493](info): commencing operation
krb5kdc: starting...
W20260925 05:05:59.605361 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:05:59 dist-test-slave-5n86 krb5kdc[20493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:05:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:05:59.845769 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43105
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:00.173110 20506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:00.173391 20506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:00.173547 20506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:00.180514 20506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:00.180675 20506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:00.180769 20506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:00.180876 20506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:00.180941 20506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:00.190464 20506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:00.191640 20506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:00.192973 20506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:00.204363 20511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:00.204401 20512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:00.204310 20514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:00.205322 20506 server_base.cc:1034] running on GCE node
I20260925 05:06:00.205971 20506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:00.207370 20506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:00.208611 20506 hybrid_clock.cc:648] HybridClock initialized: now 1790312760208577 us; error 45 us; skew 500 ppm
I20260925 05:06:00.208957 20506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:00 dist-test-slave-5n86 krb5kdc[20493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:00.217180 20506 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:06:00.218647 20506 webserver.cc:460] Webserver started at http://127.0.0.1:40269/ using document root <none> and password file <none>
I20260925 05:06:00.219296 20506 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:00.219420 20506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:00.219725 20506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:00.222774 20506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "76357f5321104b499152b8d3729d4f57"
format_stamp: "Formatted at 2026-09-25 05:06:00 on dist-test-slave-5n86"
server_key: "0e93983f3fe08948ae180bc063370d11"
server_key_iv: "1af539c869a22c4c125995345e3877ca"
server_key_version: "encryptionkey@0"
I20260925 05:06:00.223789 20506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "76357f5321104b499152b8d3729d4f57"
format_stamp: "Formatted at 2026-09-25 05:06:00 on dist-test-slave-5n86"
server_key: "0e93983f3fe08948ae180bc063370d11"
server_key_iv: "1af539c869a22c4c125995345e3877ca"
server_key_version: "encryptionkey@0"
I20260925 05:06:00.229154 20506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 05:06:00.232389 20521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:00.233724 20506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:06:00.233904 20506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "76357f5321104b499152b8d3729d4f57"
format_stamp: "Formatted at 2026-09-25 05:06:00 on dist-test-slave-5n86"
server_key: "0e93983f3fe08948ae180bc063370d11"
server_key_iv: "1af539c869a22c4c125995345e3877ca"
server_key_version: "encryptionkey@0"
I20260925 05:06:00.234093 20506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:00.239658 20506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:00.243613 20506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:00.243902 20506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:00.262320 20506 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43105
I20260925 05:06:00.262344 20572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43105 every 8 connection(s)
I20260925 05:06:00.263644 20506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:06:00.267369 20573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:00.272598 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 20506
I20260925 05:06:00.272790 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:06:00.273222 31708 external_mini_cluster.cc:1428] Setting key 24b9b21515caa362843221ea491d273b
I20260925 05:06:00.278774 20573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76357f5321104b499152b8d3729d4f57: Bootstrap starting.
I20260925 05:06:00.280791 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.274906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:36983 (local address 127.0.0.1:43105)
0925 05:06:00.275685 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.275699 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.275743 (+ 44us) server_negotiation.cc:407] Connection header received
0925 05:06:00.277478 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.277501 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.277820 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.278118 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.279214 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.279227 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.279240 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:06:00.279259 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.279325 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:06:00.279344 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.279509 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.279632 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.279812 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":375,"threads_started":1}
I20260925 05:06:00.282255 20573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76357f5321104b499152b8d3729d4f57: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:00.283442 20573 log.cc:824] T 00000000000000000000000000000000 P 76357f5321104b499152b8d3729d4f57: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:00.284327 31708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:06:00.286540 20573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76357f5321104b499152b8d3729d4f57: No bootstrap required, opened a new log
I20260925 05:06:00.292054 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.289545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:54131 (local address 127.0.0.1:43105)
0925 05:06:00.289699 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.289704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.290200 (+ 496us) server_negotiation.cc:407] Connection header received
0925 05:06:00.290348 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.290352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.290395 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.290478 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.291374 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.291378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.291381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:06:00.291395 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.291430 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:06:00.291444 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.291533 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.291670 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.291718 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 05:06:00.295536 20573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76357f5321104b499152b8d3729d4f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76357f5321104b499152b8d3729d4f57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260925 05:06:00.295864 20573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76357f5321104b499152b8d3729d4f57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:00.295958 20573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76357f5321104b499152b8d3729d4f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76357f5321104b499152b8d3729d4f57, State: Initialized, Role: FOLLOWER
I20260925 05:06:00.296577 20573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76357f5321104b499152b8d3729d4f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76357f5321104b499152b8d3729d4f57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260925 05:06:00.296814 20573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76357f5321104b499152b8d3729d4f57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:06:00.296963 20573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76357f5321104b499152b8d3729d4f57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:06:00.297128 20573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76357f5321104b499152b8d3729d4f57 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:00.299163 20573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76357f5321104b499152b8d3729d4f57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76357f5321104b499152b8d3729d4f57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260925 05:06:00.299623 20573 leader_election.cc:302] T 00000000000000000000000000000000 P 76357f5321104b499152b8d3729d4f57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76357f5321104b499152b8d3729d4f57; no voters:
I20260925 05:06:00.300132 20573 leader_election.cc:288] T 00000000000000000000000000000000 P 76357f5321104b499152b8d3729d4f57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:06:00.300278 20583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76357f5321104b499152b8d3729d4f57 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:00.300971 20583 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76357f5321104b499152b8d3729d4f57 [term 1 LEADER]: Becoming Leader. State: Replica: 76357f5321104b499152b8d3729d4f57, State: Running, Role: LEADER
I20260925 05:06:00.301549 20583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76357f5321104b499152b8d3729d4f57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76357f5321104b499152b8d3729d4f57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260925 05:06:00.302279 20573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76357f5321104b499152b8d3729d4f57 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:06:00.304682 20585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76357f5321104b499152b8d3729d4f57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76357f5321104b499152b8d3729d4f57. Latest consensus state: current_term: 1 leader_uuid: "76357f5321104b499152b8d3729d4f57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76357f5321104b499152b8d3729d4f57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } } }
I20260925 05:06:00.305176 20585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76357f5321104b499152b8d3729d4f57 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:00.305982 20584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76357f5321104b499152b8d3729d4f57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76357f5321104b499152b8d3729d4f57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76357f5321104b499152b8d3729d4f57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } } }
I20260925 05:06:00.306380 20584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76357f5321104b499152b8d3729d4f57 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:00.306706 20591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:06:00.312335 20591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:06:00.313061 20600 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 05:06:00.320430 20591 catalog_manager.cc:1347] Generated new cluster ID: da4850f0467743c29102a9f535c1856d
I20260925 05:06:00.320549 20591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:06:00.369467 20591 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:06:00.369907 20591 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:06:00.389896 20591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76357f5321104b499152b8d3729d4f57: Generated new TSK 0
I20260925 05:06:00.390863 20591 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:06:00.395839 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 20506
W20260925 05:06:00.396854 20568 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.152:36983
W20260925 05:06:00.397532 31708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 20506: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:06:00.397677 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 20506
2026-09-25T05:06:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1874 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:06:00 dist-test-slave-5n86 krb5kdc[20604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:06:00 dist-test-slave-5n86 krb5kdc[20604](info): set up 2 sockets
Sep 25 05:06:00 dist-test-slave-5n86 krb5kdc[20604](info): commencing operation
krb5kdc: starting...
W20260925 05:06:01.501294 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:06:01 dist-test-slave-5n86 krb5kdc[20604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:06:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:06:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:06:01.745757 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32809
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:02.064811 20617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:02.065078 20617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:02.065237 20617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:02.072196 20617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:02.072340 20617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:02.072448 20617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:02.072567 20617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:02.072656 20617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:02.082075 20617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32809
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:02.083333 20617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:02.084654 20617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:02.096515 20625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:02.096602 20623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:02.096853 20622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:02.097401 20617 server_base.cc:1034] running on GCE node
I20260925 05:06:02.098080 20617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:02.099409 20617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:02.100643 20617 hybrid_clock.cc:648] HybridClock initialized: now 1790312762100606 us; error 30 us; skew 500 ppm
I20260925 05:06:02.100993 20617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:02 dist-test-slave-5n86 krb5kdc[20604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:02.104121 20617 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:06:02.105737 20617 webserver.cc:460] Webserver started at http://127.0.0.1:37797/ using document root <none> and password file <none>
I20260925 05:06:02.106467 20617 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:02.106611 20617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:02.106910 20617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:02.109894 20617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "b8b0f3c0f9984d13b0ae93a0c24b3367"
format_stamp: "Formatted at 2026-09-25 05:06:02 on dist-test-slave-5n86"
server_key: "a7bc317b4dbad7cbaa451b05ec2cf4a5"
server_key_iv: "7587d355153518271d72a565b220f231"
server_key_version: "encryptionkey@0"
I20260925 05:06:02.110770 20617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "b8b0f3c0f9984d13b0ae93a0c24b3367"
format_stamp: "Formatted at 2026-09-25 05:06:02 on dist-test-slave-5n86"
server_key: "a7bc317b4dbad7cbaa451b05ec2cf4a5"
server_key_iv: "7587d355153518271d72a565b220f231"
server_key_version: "encryptionkey@0"
I20260925 05:06:02.115770 20617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 05:06:02.119289 20632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:02.120636 20617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:06:02.120813 20617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "b8b0f3c0f9984d13b0ae93a0c24b3367"
format_stamp: "Formatted at 2026-09-25 05:06:02 on dist-test-slave-5n86"
server_key: "a7bc317b4dbad7cbaa451b05ec2cf4a5"
server_key_iv: "7587d355153518271d72a565b220f231"
server_key_version: "encryptionkey@0"
I20260925 05:06:02.120997 20617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:02.138942 20617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:02.142614 20617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:02.142884 20617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:02.161203 20617 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32809
I20260925 05:06:02.161242 20683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32809 every 8 connection(s)
I20260925 05:06:02.162501 20617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:06:02.166014 20684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:02.167212 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 20617
I20260925 05:06:02.167404 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:06:02.167872 31708 external_mini_cluster.cc:1428] Setting key 8d961b516790fde1806f312fc606de8f
I20260925 05:06:02.179800 20684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8b0f3c0f9984d13b0ae93a0c24b3367: Bootstrap starting.
Sep 25 05:06:02 dist-test-slave-5n86 krb5kdc[20604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 05:06:02.182746 20684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8b0f3c0f9984d13b0ae93a0c24b3367: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:02.183934 20684 log.cc:824] T 00000000000000000000000000000000 P b8b0f3c0f9984d13b0ae93a0c24b3367: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:02.187188 20684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8b0f3c0f9984d13b0ae93a0c24b3367: No bootstrap required, opened a new log
I20260925 05:06:02.191830 20687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:02.169668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:60757 (local address 127.0.0.1:32809)
0925 05:06:02.170456 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:02.170475 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:02.170526 (+ 51us) server_negotiation.cc:407] Connection header received
0925 05:06:02.172430 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:02.172457 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:02.172989 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:02.173416 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:02.174865 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:02.176937 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:02.178214 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:02.179200 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:02.179375 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:02.183390 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:02.183400 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:02.183417 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:02.183458 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:02.187198 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:02.187914 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:02.187921 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:02.187931 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:02.188114 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:02.188630 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:02.188635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:02.188638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:02.189304 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:02.189652 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:02.190237 (+ 585us) server_negotiation.cc:299] Negotiation successful
0925 05:06:02.190564 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":306,"threads_started":1}
I20260925 05:06:02.195538 31708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:06:02.196211 20684 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8b0f3c0f9984d13b0ae93a0c24b3367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b0f3c0f9984d13b0ae93a0c24b3367" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32809 } }
I20260925 05:06:02.196610 20684 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8b0f3c0f9984d13b0ae93a0c24b3367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:02.196705 20684 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8b0f3c0f9984d13b0ae93a0c24b3367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8b0f3c0f9984d13b0ae93a0c24b3367, State: Initialized, Role: FOLLOWER
I20260925 05:06:02.197275 20684 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8b0f3c0f9984d13b0ae93a0c24b3367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b0f3c0f9984d13b0ae93a0c24b3367" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32809 } }
I20260925 05:06:02.197531 20684 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8b0f3c0f9984d13b0ae93a0c24b3367 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:06:02.197647 20684 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8b0f3c0f9984d13b0ae93a0c24b3367 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:06:02.197800 20684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8b0f3c0f9984d13b0ae93a0c24b3367 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:02.200944 20684 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8b0f3c0f9984d13b0ae93a0c24b3367 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b0f3c0f9984d13b0ae93a0c24b3367" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32809 } }
I20260925 05:06:02.201534 20684 leader_election.cc:302] T 00000000000000000000000000000000 P b8b0f3c0f9984d13b0ae93a0c24b3367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8b0f3c0f9984d13b0ae93a0c24b3367; no voters:
I20260925 05:06:02.202180 20684 leader_election.cc:288] T 00000000000000000000000000000000 P b8b0f3c0f9984d13b0ae93a0c24b3367 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:06:02.202301 20694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8b0f3c0f9984d13b0ae93a0c24b3367 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:02.203783 20694 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8b0f3c0f9984d13b0ae93a0c24b3367 [term 1 LEADER]: Becoming Leader. State: Replica: b8b0f3c0f9984d13b0ae93a0c24b3367, State: Running, Role: LEADER
I20260925 05:06:02.204414 20684 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8b0f3c0f9984d13b0ae93a0c24b3367 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:06:02.204320 20694 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8b0f3c0f9984d13b0ae93a0c24b3367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b0f3c0f9984d13b0ae93a0c24b3367" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32809 } }
I20260925 05:06:02.209571 20695 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b0f3c0f9984d13b0ae93a0c24b3367 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8b0f3c0f9984d13b0ae93a0c24b3367" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b0f3c0f9984d13b0ae93a0c24b3367" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32809 } } }
I20260925 05:06:02.209947 20695 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b0f3c0f9984d13b0ae93a0c24b3367 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:02.212343 20696 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b0f3c0f9984d13b0ae93a0c24b3367 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8b0f3c0f9984d13b0ae93a0c24b3367. Latest consensus state: current_term: 1 leader_uuid: "b8b0f3c0f9984d13b0ae93a0c24b3367" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b0f3c0f9984d13b0ae93a0c24b3367" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32809 } } }
I20260925 05:06:02.212669 20696 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b0f3c0f9984d13b0ae93a0c24b3367 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:02.218883 20701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:06:02.220278 20687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:02.200518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:59791 (local address 127.0.0.1:32809)
0925 05:06:02.200736 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:02.200745 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:02.201166 (+ 421us) server_negotiation.cc:407] Connection header received
0925 05:06:02.201324 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:02.201329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:02.201383 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:02.201459 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:02.203346 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:02.204747 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:02.206740 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:02.207633 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:02.207804 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:02.211768 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:02.211787 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:02.211795 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:02.211843 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:02.216284 (+ 4441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:02.217528 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:02.217541 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:02.217547 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:02.217724 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:02.218432 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:02.218437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:02.218440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:02.219002 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:02.219277 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:02.219708 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 05:06:02.219879 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 05:06:02.225019 20701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:06:02.234668 20701 catalog_manager.cc:1347] Generated new cluster ID: 173ef2525a8745b19339a9a0ba6635b4
I20260925 05:06:02.234783 20701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:06:02.255425 20701 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:06:02.256774 20701 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:06:02.263468 20701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8b0f3c0f9984d13b0ae93a0c24b3367: Generated new TSK 0
I20260925 05:06:02.264206 20701 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:06:02.274281 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 20617
I20260925 05:06:02.275364 20679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:06:02.366266 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 20617
W20260925 05:06:02.382908 20602 connection.cc:535] client connection to 127.0.0.1:32809 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32809): Connection reset by peer (error 104)
2026-09-25T05:06:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1968 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:06:02 dist-test-slave-5n86 krb5kdc[20717](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:06:02 dist-test-slave-5n86 krb5kdc[20717](info): set up 2 sockets
Sep 25 05:06:02 dist-test-slave-5n86 krb5kdc[20717](info): commencing operation
krb5kdc: starting...
W20260925 05:06:03.477190 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:06:03 dist-test-slave-5n86 krb5kdc[20717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:06:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:06:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:06:03.719614 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45941
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:04.042042 20730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:04.042311 20730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:04.042402 20730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:04.049372 20730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:04.049468 20730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:04.049515 20730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:04.049573 20730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:04.049616 20730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:04.059074 20730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45941
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:04.060227 20730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:04.061457 20730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:04.072760 20735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:04.072793 20738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:04.072850 20736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:04.073679 20730 server_base.cc:1034] running on GCE node
I20260925 05:06:04.074368 20730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:04.075598 20730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:04.076817 20730 hybrid_clock.cc:648] HybridClock initialized: now 1790312764076776 us; error 36 us; skew 500 ppm
I20260925 05:06:04.077180 20730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:04 dist-test-slave-5n86 krb5kdc[20717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:04.080207 20730 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:06:04.081704 20730 webserver.cc:460] Webserver started at http://127.0.0.1:40195/ using document root <none> and password file <none>
I20260925 05:06:04.082358 20730 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:04.082489 20730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:04.082785 20730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:04.085669 20730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "923da31013524be09be496f60c2caf77"
format_stamp: "Formatted at 2026-09-25 05:06:04 on dist-test-slave-5n86"
server_key: "ffc3771d29052f8ccafe1bb67da536ae"
server_key_iv: "125e128226348dd2247679afe5fa454f"
server_key_version: "encryptionkey@0"
I20260925 05:06:04.086413 20730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "923da31013524be09be496f60c2caf77"
format_stamp: "Formatted at 2026-09-25 05:06:04 on dist-test-slave-5n86"
server_key: "ffc3771d29052f8ccafe1bb67da536ae"
server_key_iv: "125e128226348dd2247679afe5fa454f"
server_key_version: "encryptionkey@0"
I20260925 05:06:04.091274 20730 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 05:06:04.094632 20745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:04.095925 20730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:06:04.096110 20730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "923da31013524be09be496f60c2caf77"
format_stamp: "Formatted at 2026-09-25 05:06:04 on dist-test-slave-5n86"
server_key: "ffc3771d29052f8ccafe1bb67da536ae"
server_key_iv: "125e128226348dd2247679afe5fa454f"
server_key_version: "encryptionkey@0"
I20260925 05:06:04.096304 20730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:04.119765 20730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:04.123674 20730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:04.123955 20730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:04.142128 20730 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45941
I20260925 05:06:04.142168 20796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45941 every 8 connection(s)
I20260925 05:06:04.143414 20730 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:06:04.147078 20797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:04.150847 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 20730
I20260925 05:06:04.151041 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:06:04.151599 31708 external_mini_cluster.cc:1428] Setting key d5e95d37032f05a6e0d4319c578f1c84
I20260925 05:06:04.159551 20797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 923da31013524be09be496f60c2caf77: Bootstrap starting.
I20260925 05:06:04.163081 20797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 923da31013524be09be496f60c2caf77: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:04.164211 20797 log.cc:824] T 00000000000000000000000000000000 P 923da31013524be09be496f60c2caf77: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:04.164341 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.153394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:36851 (local address 127.0.0.1:45941)
0925 05:06:04.154113 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:04.154123 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:04.154162 (+ 39us) server_negotiation.cc:407] Connection header received
0925 05:06:04.155718 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:04.155739 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:04.156126 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:04.156436 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:04.157005 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:04.158272 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:04.160080 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:04.160785 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:04.160915 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:04.162204 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:04.162211 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:04.162223 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:06:04.162714 (+ 491us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:04.162777 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:06:04.162802 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:04.162913 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:04.163238 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 05:06:04.163386 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":322,"threads_started":1}
I20260925 05:06:04.167042 20797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 923da31013524be09be496f60c2caf77: No bootstrap required, opened a new log
I20260925 05:06:04.167912 31708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:06:04.174989 20797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 923da31013524be09be496f60c2caf77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "923da31013524be09be496f60c2caf77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45941 } }
I20260925 05:06:04.175354 20797 raft_consensus.cc:374] T 00000000000000000000000000000000 P 923da31013524be09be496f60c2caf77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:04.175465 20797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 923da31013524be09be496f60c2caf77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923da31013524be09be496f60c2caf77, State: Initialized, Role: FOLLOWER
I20260925 05:06:04.176024 20797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 923da31013524be09be496f60c2caf77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "923da31013524be09be496f60c2caf77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45941 } }
I20260925 05:06:04.176281 20797 raft_consensus.cc:388] T 00000000000000000000000000000000 P 923da31013524be09be496f60c2caf77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:06:04.176450 20797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 923da31013524be09be496f60c2caf77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:06:04.176635 20797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 923da31013524be09be496f60c2caf77 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:04.178128 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.171970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:51197 (local address 127.0.0.1:45941)
0925 05:06:04.172106 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:04.172110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:04.172612 (+ 502us) server_negotiation.cc:407] Connection header received
0925 05:06:04.172808 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:04.172813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:04.172875 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:04.172982 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:04.173468 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:04.174199 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:04.175574 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:04.176083 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:04.176220 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:04.177406 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:04.177410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:04.177413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:06:04.177428 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:04.177485 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:06:04.177500 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:04.177626 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:04.177811 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 05:06:04.177866 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 05:06:04.179225 20797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 923da31013524be09be496f60c2caf77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "923da31013524be09be496f60c2caf77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45941 } }
I20260925 05:06:04.179713 20797 leader_election.cc:302] T 00000000000000000000000000000000 P 923da31013524be09be496f60c2caf77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 923da31013524be09be496f60c2caf77; no voters:
I20260925 05:06:04.180202 20797 leader_election.cc:288] T 00000000000000000000000000000000 P 923da31013524be09be496f60c2caf77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:06:04.180341 20807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 923da31013524be09be496f60c2caf77 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:04.181005 20807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 923da31013524be09be496f60c2caf77 [term 1 LEADER]: Becoming Leader. State: Replica: 923da31013524be09be496f60c2caf77, State: Running, Role: LEADER
I20260925 05:06:04.181505 20807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 923da31013524be09be496f60c2caf77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "923da31013524be09be496f60c2caf77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45941 } }
I20260925 05:06:04.182180 20797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 923da31013524be09be496f60c2caf77 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:06:04.184108 20808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 923da31013524be09be496f60c2caf77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "923da31013524be09be496f60c2caf77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "923da31013524be09be496f60c2caf77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45941 } } }
I20260925 05:06:04.184476 20808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 923da31013524be09be496f60c2caf77 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:04.186662 20809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 923da31013524be09be496f60c2caf77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 923da31013524be09be496f60c2caf77. Latest consensus state: current_term: 1 leader_uuid: "923da31013524be09be496f60c2caf77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "923da31013524be09be496f60c2caf77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45941 } } }
I20260925 05:06:04.187088 20809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 923da31013524be09be496f60c2caf77 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:04.187276 20813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:06:04.192196 20813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:06:04.200770 20813 catalog_manager.cc:1347] Generated new cluster ID: 36184e282ddb43959efab8ae9073d771
I20260925 05:06:04.200909 20813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:06:04.212517 20813 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:06:04.214227 20813 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:06:04.220830 20813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 923da31013524be09be496f60c2caf77: Generated new TSK 0
I20260925 05:06:04.221562 20813 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:06:04.236121 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 20730
W20260925 05:06:04.237553 20792 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.152:36851
W20260925 05:06:04.238251 31708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 20730: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:06:04.238387 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 20730
W20260925 05:06:04.255985 20715 connection.cc:535] client connection to 127.0.0.1:45941 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45941): Connection reset by peer (error 104)
2026-09-25T05:06:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1872 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:06:04 dist-test-slave-5n86 krb5kdc[20828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:06:04 dist-test-slave-5n86 krb5kdc[20828](info): set up 2 sockets
Sep 25 05:06:04 dist-test-slave-5n86 krb5kdc[20828](info): commencing operation
krb5kdc: starting...
W20260925 05:06:05.345268 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:06:05 dist-test-slave-5n86 krb5kdc[20828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:06:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:06:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:06:05.575919 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40551
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:05.888806 20841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:05.889115 20841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:05.889278 20841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:05.896289 20841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:05.896443 20841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:05.896541 20841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:05.896677 20841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:05.896754 20841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:05.906152 20841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40551
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312430481417-31708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:05.907384 20841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:05.908710 20841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:05.920142 20847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:05.920470 20846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:05.920965 20849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:05.920984 20841 server_base.cc:1034] running on GCE node
I20260925 05:06:05.921685 20841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:05.922983 20841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:05.924213 20841 hybrid_clock.cc:648] HybridClock initialized: now 1790312765924177 us; error 55 us; skew 500 ppm
I20260925 05:06:05.924657 20841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:05 dist-test-slave-5n86 krb5kdc[20828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:05.933145 20841 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:06:05.934631 20841 webserver.cc:460] Webserver started at http://127.0.0.1:46805/ using document root <none> and password file <none>
I20260925 05:06:05.935297 20841 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:05.935441 20841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:05.935737 20841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:05.938618 20841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "1a7340c626f04ae39c98c73a9d043763"
format_stamp: "Formatted at 2026-09-25 05:06:05 on dist-test-slave-5n86"
server_key: "94e273a489e91fe2dd850b0ab8505078"
server_key_iv: "152f37d06aff05837491ac62b761fe9e"
server_key_version: "encryptionkey@0"
I20260925 05:06:05.939371 20841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "1a7340c626f04ae39c98c73a9d043763"
format_stamp: "Formatted at 2026-09-25 05:06:05 on dist-test-slave-5n86"
server_key: "94e273a489e91fe2dd850b0ab8505078"
server_key_iv: "152f37d06aff05837491ac62b761fe9e"
server_key_version: "encryptionkey@0"
I20260925 05:06:05.944314 20841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 05:06:05.947631 20856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:05.948869 20841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:06:05.949036 20841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "1a7340c626f04ae39c98c73a9d043763"
format_stamp: "Formatted at 2026-09-25 05:06:05 on dist-test-slave-5n86"
server_key: "94e273a489e91fe2dd850b0ab8505078"
server_key_iv: "152f37d06aff05837491ac62b761fe9e"
server_key_version: "encryptionkey@0"
I20260925 05:06:05.949266 20841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:05.954552 20841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:05.958601 20841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:05.958886 20841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:05.981364 20841 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40551
I20260925 05:06:05.981398 20907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40551 every 8 connection(s)
I20260925 05:06:05.982806 20841 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:06:05.985412 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 20841
I20260925 05:06:05.985579 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:06:05.986023 31708 external_mini_cluster.cc:1428] Setting key bec8598ea3c335c8f7af2120927a7a52
I20260925 05:06:05.986586 20908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:05.993271 20911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:05.987712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:44859 (local address 127.0.0.1:40551)
0925 05:06:05.988416 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:05.988427 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:05.988464 (+ 37us) server_negotiation.cc:407] Connection header received
0925 05:06:05.990005 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:05.990025 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:05.990336 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:05.990714 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:05.991698 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:05.991704 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:05.991716 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:06:05.991728 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:05.991792 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:06:05.991811 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:05.992043 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:05.992180 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 05:06:05.992361 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":299,"threads_started":1}
I20260925 05:06:05.996690 31708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:06:05.999526 20908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a7340c626f04ae39c98c73a9d043763: Bootstrap starting.
I20260925 05:06:06.002852 20908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a7340c626f04ae39c98c73a9d043763: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:06.003188 20911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.000738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:52865 (local address 127.0.0.1:40551)
0925 05:06:06.000874 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:06.000878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:06.001290 (+ 412us) server_negotiation.cc:407] Connection header received
0925 05:06:06.001474 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:06.001479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:06.001536 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:06.001631 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:06.002519 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:06.002524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:06.002528 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:06:06.002551 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:06.002585 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:06:06.002599 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:06.002708 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:06.002825 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 05:06:06.002894 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 05:06:06.004025 20908 log.cc:824] T 00000000000000000000000000000000 P 1a7340c626f04ae39c98c73a9d043763: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:06.006517 20908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a7340c626f04ae39c98c73a9d043763: No bootstrap required, opened a new log
I20260925 05:06:06.013827 20908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a7340c626f04ae39c98c73a9d043763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a7340c626f04ae39c98c73a9d043763" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40551 } }
I20260925 05:06:06.014173 20908 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a7340c626f04ae39c98c73a9d043763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:06.014277 20908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a7340c626f04ae39c98c73a9d043763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a7340c626f04ae39c98c73a9d043763, State: Initialized, Role: FOLLOWER
I20260925 05:06:06.014845 20908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a7340c626f04ae39c98c73a9d043763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a7340c626f04ae39c98c73a9d043763" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40551 } }
I20260925 05:06:06.015089 20908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a7340c626f04ae39c98c73a9d043763 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:06:06.015223 20908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a7340c626f04ae39c98c73a9d043763 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:06:06.015389 20908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a7340c626f04ae39c98c73a9d043763 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:06.017481 20908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a7340c626f04ae39c98c73a9d043763 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a7340c626f04ae39c98c73a9d043763" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40551 } }
I20260925 05:06:06.017975 20908 leader_election.cc:302] T 00000000000000000000000000000000 P 1a7340c626f04ae39c98c73a9d043763 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a7340c626f04ae39c98c73a9d043763; no voters:
I20260925 05:06:06.018651 20908 leader_election.cc:288] T 00000000000000000000000000000000 P 1a7340c626f04ae39c98c73a9d043763 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:06:06.018815 20918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a7340c626f04ae39c98c73a9d043763 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:06.019532 20918 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a7340c626f04ae39c98c73a9d043763 [term 1 LEADER]: Becoming Leader. State: Replica: 1a7340c626f04ae39c98c73a9d043763, State: Running, Role: LEADER
I20260925 05:06:06.020125 20918 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a7340c626f04ae39c98c73a9d043763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a7340c626f04ae39c98c73a9d043763" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40551 } }
I20260925 05:06:06.020633 20908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a7340c626f04ae39c98c73a9d043763 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:06:06.022970 20919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a7340c626f04ae39c98c73a9d043763 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a7340c626f04ae39c98c73a9d043763" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a7340c626f04ae39c98c73a9d043763" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40551 } } }
I20260925 05:06:06.023434 20919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a7340c626f04ae39c98c73a9d043763 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:06.023350 20920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a7340c626f04ae39c98c73a9d043763 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a7340c626f04ae39c98c73a9d043763. Latest consensus state: current_term: 1 leader_uuid: "1a7340c626f04ae39c98c73a9d043763" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a7340c626f04ae39c98c73a9d043763" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40551 } } }
I20260925 05:06:06.023665 20920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a7340c626f04ae39c98c73a9d043763 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:06.025449 20927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:06:06.029934 20927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:06:06.038437 20927 catalog_manager.cc:1347] Generated new cluster ID: 956959a0cf9740ff85888d3c821a1921
I20260925 05:06:06.038537 20927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:06:06.064810 20927 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:06:06.065248 20927 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:06:06.075449 20927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a7340c626f04ae39c98c73a9d043763: Generated new TSK 0
I20260925 05:06:06.076139 20927 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:06:06.083566 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 20841
W20260925 05:06:06.084677 20903 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.152:44859
W20260925 05:06:06.085340 31708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 20841: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:06:06.085485 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 20841
2026-09-25T05:06:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1845 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:06:06 dist-test-slave-5n86 krb5kdc[20939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:06:06 dist-test-slave-5n86 krb5kdc[20939](info): set up 2 sockets
Sep 25 05:06:06 dist-test-slave-5n86 krb5kdc[20939](info): commencing operation
krb5kdc: starting...
W20260925 05:06:07.177088 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:06:07 dist-test-slave-5n86 krb5kdc[20939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:06:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:06:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:06:07.415910 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:44987
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:41475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:44987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:07.736951 20952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:07.737231 20952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:07.737332 20952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:07.744297 20952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:07.744398 20952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:07.744444 20952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:07.744503 20952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:07.744563 20952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:07.754004 20952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:41475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:44987
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:44987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:07.755133 20952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:07.756333 20952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:07.767459 20957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:07.767561 20958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:07.767661 20960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:07.768635 20952 server_base.cc:1034] running on GCE node
I20260925 05:06:07.769384 20952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:07.770820 20952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:07.772042 20952 hybrid_clock.cc:648] HybridClock initialized: now 1790312767771957 us; error 74 us; skew 500 ppm
I20260925 05:06:07.772394 20952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:07 dist-test-slave-5n86 krb5kdc[20939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312767, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:07.775677 20952 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:06:07.777048 20952 webserver.cc:460] Webserver started at http://127.30.247.62:43161/ using document root <none> and password file <none>
I20260925 05:06:07.777669 20952 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:07.777766 20952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:07.778012 20952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:07.780942 20952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "02559efd1d934b16bf49ec13618f0a28"
format_stamp: "Formatted at 2026-09-25 05:06:07 on dist-test-slave-5n86"
server_key: "beac6d40909230149670534982396f15"
server_key_iv: "3c3a5c81f44491ddd0df1a467eab1f5d"
server_key_version: "encryptionkey@0"
I20260925 05:06:07.781622 20952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "02559efd1d934b16bf49ec13618f0a28"
format_stamp: "Formatted at 2026-09-25 05:06:07 on dist-test-slave-5n86"
server_key: "beac6d40909230149670534982396f15"
server_key_iv: "3c3a5c81f44491ddd0df1a467eab1f5d"
server_key_version: "encryptionkey@0"
I20260925 05:06:07.786428 20952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 05:06:07.789721 20967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:07.790954 20952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:06:07.791128 20952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "02559efd1d934b16bf49ec13618f0a28"
format_stamp: "Formatted at 2026-09-25 05:06:07 on dist-test-slave-5n86"
server_key: "beac6d40909230149670534982396f15"
server_key_iv: "3c3a5c81f44491ddd0df1a467eab1f5d"
server_key_version: "encryptionkey@0"
I20260925 05:06:07.791294 20952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:07.813634 20952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:07.817623 20952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:07.817899 20952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:07.836421 20952 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:44987
I20260925 05:06:07.836454 21018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:44987 every 8 connection(s)
I20260925 05:06:07.837759 20952 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:06:07.841019 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 20952
I20260925 05:06:07.841188 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:06:07.841440 21019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:07.841666 31708 external_mini_cluster.cc:1428] Setting key 9486476abab81a3ebc5a7963a813453f
I20260925 05:06:07.855552 21019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02559efd1d934b16bf49ec13618f0a28: Bootstrap starting.
Sep 25 05:06:07 dist-test-slave-5n86 krb5kdc[20939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:06:07.859329 21019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02559efd1d934b16bf49ec13618f0a28: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:07.860736 21019 log.cc:824] T 00000000000000000000000000000000 P 02559efd1d934b16bf49ec13618f0a28: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:07.864121 21019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02559efd1d934b16bf49ec13618f0a28: No bootstrap required, opened a new log
I20260925 05:06:07.865991 21022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:07.843404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54709 (local address 127.30.247.62:44987)
0925 05:06:07.844211 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:07.844225 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:07.844278 (+ 53us) server_negotiation.cc:407] Connection header received
0925 05:06:07.846335 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:07.846363 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:07.846876 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:07.847314 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:07.848564 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:07.850547 (+ 1983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:07.851789 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:07.852916 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:07.853107 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:07.857240 (+ 4133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:07.857264 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:07.857291 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:07.857346 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:07.861523 (+ 4177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:07.862432 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:07.862440 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:07.862448 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:07.862648 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:07.863096 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:07.863101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:07.863104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:07.863761 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:07.864067 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:07.864663 (+ 596us) server_negotiation.cc:299] Negotiation successful
0925 05:06:07.865001 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":331,"threads_started":1}
I20260925 05:06:07.869796 31708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:06:07.872181 21019 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02559efd1d934b16bf49ec13618f0a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02559efd1d934b16bf49ec13618f0a28" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44987 } }
I20260925 05:06:07.872608 21019 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02559efd1d934b16bf49ec13618f0a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:07.872728 21019 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02559efd1d934b16bf49ec13618f0a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02559efd1d934b16bf49ec13618f0a28, State: Initialized, Role: FOLLOWER
I20260925 05:06:07.873314 21019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02559efd1d934b16bf49ec13618f0a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02559efd1d934b16bf49ec13618f0a28" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44987 } }
I20260925 05:06:07.873569 21019 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02559efd1d934b16bf49ec13618f0a28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:06:07.873718 21019 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02559efd1d934b16bf49ec13618f0a28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:06:07.873908 21019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02559efd1d934b16bf49ec13618f0a28 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:07.876353 21019 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02559efd1d934b16bf49ec13618f0a28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02559efd1d934b16bf49ec13618f0a28" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44987 } }
I20260925 05:06:07.876924 21019 leader_election.cc:302] T 00000000000000000000000000000000 P 02559efd1d934b16bf49ec13618f0a28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02559efd1d934b16bf49ec13618f0a28; no voters:
I20260925 05:06:07.877579 21019 leader_election.cc:288] T 00000000000000000000000000000000 P 02559efd1d934b16bf49ec13618f0a28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:06:07.877722 21029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02559efd1d934b16bf49ec13618f0a28 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:07.878854 21029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02559efd1d934b16bf49ec13618f0a28 [term 1 LEADER]: Becoming Leader. State: Replica: 02559efd1d934b16bf49ec13618f0a28, State: Running, Role: LEADER
I20260925 05:06:07.879416 21029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02559efd1d934b16bf49ec13618f0a28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02559efd1d934b16bf49ec13618f0a28" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44987 } }
I20260925 05:06:07.879973 21019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02559efd1d934b16bf49ec13618f0a28 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:06:07.882705 21030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02559efd1d934b16bf49ec13618f0a28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02559efd1d934b16bf49ec13618f0a28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02559efd1d934b16bf49ec13618f0a28" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44987 } } }
I20260925 05:06:07.883230 21030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02559efd1d934b16bf49ec13618f0a28 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:07.882659 21031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02559efd1d934b16bf49ec13618f0a28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02559efd1d934b16bf49ec13618f0a28. Latest consensus state: current_term: 1 leader_uuid: "02559efd1d934b16bf49ec13618f0a28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02559efd1d934b16bf49ec13618f0a28" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 44987 } } }
I20260925 05:06:07.883388 21031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02559efd1d934b16bf49ec13618f0a28 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:07.885407 21035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:06:07.890317 21035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:06:07.896625 21022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:07.874036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58819 (local address 127.30.247.62:44987)
0925 05:06:07.874211 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:07.874219 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:07.874707 (+ 488us) server_negotiation.cc:407] Connection header received
0925 05:06:07.874823 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:07.874828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:07.874917 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:07.875020 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:07.876663 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:07.877541 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:07.879269 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:07.879878 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:07.880025 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:07.885889 (+ 5864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:07.885899 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:07.885907 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:07.885961 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:07.890920 (+ 4959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:07.892220 (+ 1300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:07.892245 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:07.892255 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:07.892448 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:07.893163 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:07.893171 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:07.893177 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:07.893797 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:07.894149 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:07.895822 (+ 1673us) server_negotiation.cc:299] Negotiation successful
0925 05:06:07.896075 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 05:06:07.906167 21035 catalog_manager.cc:1347] Generated new cluster ID: 76dde23773cc4343bc72d4bb594e5d6f
I20260925 05:06:07.906283 21035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:06:07.934707 21035 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:06:07.936488 21035 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:06:07.951700 21035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02559efd1d934b16bf49ec13618f0a28: Generated new TSK 0
I20260925 05:06:07.952754 21035 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:06:07.966809 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 20952
I20260925 05:06:07.968180 21014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:06:08.061720 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 20952
W20260925 05:06:08.078644 20937 connection.cc:535] client connection to 127.30.247.62:44987 recv error: Network error: failed to read from TLS socket (remote: 127.30.247.62:44987): Connection reset by peer (error 104)
2026-09-25T05:06:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1976 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:06:08 dist-test-slave-5n86 krb5kdc[21051](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:06:08 dist-test-slave-5n86 krb5kdc[21051](info): set up 2 sockets
Sep 25 05:06:08 dist-test-slave-5n86 krb5kdc[21051](info): commencing operation
krb5kdc: starting...
W20260925 05:06:09.169173 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:06:09 dist-test-slave-5n86 krb5kdc[21051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:06:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:06:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:06:09.400319 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:37893
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:37893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:09.713603 21064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:09.713946 21064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:09.714125 21064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:09.721138 21064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:09.721282 21064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:09.721398 21064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:09.721510 21064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:09.721602 21064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:09.731204 21064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:37893
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:37893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:09.732481 21064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:09.733785 21064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:09.745371 21070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:09.745395 21069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:09.745654 21072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:09.746238 21064 server_base.cc:1034] running on GCE node
I20260925 05:06:09.746950 21064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:09.748224 21064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:09.749410 21064 hybrid_clock.cc:648] HybridClock initialized: now 1790312769749362 us; error 47 us; skew 500 ppm
I20260925 05:06:09.749699 21064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:09 dist-test-slave-5n86 krb5kdc[21051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312769, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:09.752672 21064 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:06:09.754000 21064 webserver.cc:460] Webserver started at http://127.30.247.62:33293/ using document root <none> and password file <none>
I20260925 05:06:09.754652 21064 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:09.754798 21064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:09.755085 21064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:09.757946 21064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "7170713ab7ec4a419935ab4bfe2bdade"
format_stamp: "Formatted at 2026-09-25 05:06:09 on dist-test-slave-5n86"
server_key: "da3795756aa3339bb86374863c088460"
server_key_iv: "8627d4012e573fff10df31429f9d6032"
server_key_version: "encryptionkey@0"
I20260925 05:06:09.758697 21064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "7170713ab7ec4a419935ab4bfe2bdade"
format_stamp: "Formatted at 2026-09-25 05:06:09 on dist-test-slave-5n86"
server_key: "da3795756aa3339bb86374863c088460"
server_key_iv: "8627d4012e573fff10df31429f9d6032"
server_key_version: "encryptionkey@0"
I20260925 05:06:09.763561 21064 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260925 05:06:09.766928 21079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:09.768216 21064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:06:09.768427 21064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "7170713ab7ec4a419935ab4bfe2bdade"
format_stamp: "Formatted at 2026-09-25 05:06:09 on dist-test-slave-5n86"
server_key: "da3795756aa3339bb86374863c088460"
server_key_iv: "8627d4012e573fff10df31429f9d6032"
server_key_version: "encryptionkey@0"
I20260925 05:06:09.768671 21064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:09.785429 21064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:09.789219 21064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:09.789490 21064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:09.807190 21064 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:37893
I20260925 05:06:09.807219 21130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:37893 every 8 connection(s)
I20260925 05:06:09.808466 21064 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:06:09.811162 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 21064
I20260925 05:06:09.811342 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:06:09.811771 31708 external_mini_cluster.cc:1428] Setting key f01dbf5f408919b192495eac1622ae4a
I20260925 05:06:09.812347 21131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:09.826742 21134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:09.813597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56209 (local address 127.30.247.62:37893)
0925 05:06:09.814339 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:09.814353 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:09.814399 (+ 46us) server_negotiation.cc:407] Connection header received
0925 05:06:09.816910 (+ 2511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:09.816939 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:09.817451 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:09.817878 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:09.818385 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:09.820369 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:09.821702 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:09.822717 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:09.822893 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:09.824075 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:09.824087 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:09.824109 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:06:09.824853 (+ 744us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:09.824950 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:06:09.824984 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:09.825140 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:09.825380 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 05:06:09.825535 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":317,"threads_started":1}
I20260925 05:06:09.827137 21131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7170713ab7ec4a419935ab4bfe2bdade: Bootstrap starting.
I20260925 05:06:09.830384 21131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7170713ab7ec4a419935ab4bfe2bdade: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:09.830585 31708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:06:09.831390 21131 log.cc:824] T 00000000000000000000000000000000 P 7170713ab7ec4a419935ab4bfe2bdade: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:09.834007 21131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7170713ab7ec4a419935ab4bfe2bdade: No bootstrap required, opened a new log
I20260925 05:06:09.841728 21134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:09.835332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52595 (local address 127.30.247.62:37893)
0925 05:06:09.835513 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:09.835523 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:09.835918 (+ 395us) server_negotiation.cc:407] Connection header received
0925 05:06:09.836059 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:09.836065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:09.836138 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:09.836243 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:09.836756 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:09.837893 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:09.839137 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:09.839831 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:09.839961 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:09.840975 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:09.840983 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:09.840990 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:06:09.841016 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:09.841076 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:06:09.841096 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:09.841210 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:09.841346 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 05:06:09.841391 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:06:09.843577 21131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7170713ab7ec4a419935ab4bfe2bdade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7170713ab7ec4a419935ab4bfe2bdade" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37893 } }
I20260925 05:06:09.843884 21131 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7170713ab7ec4a419935ab4bfe2bdade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:09.843957 21131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7170713ab7ec4a419935ab4bfe2bdade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7170713ab7ec4a419935ab4bfe2bdade, State: Initialized, Role: FOLLOWER
I20260925 05:06:09.844391 21131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7170713ab7ec4a419935ab4bfe2bdade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7170713ab7ec4a419935ab4bfe2bdade" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37893 } }
I20260925 05:06:09.844623 21131 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7170713ab7ec4a419935ab4bfe2bdade [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:06:09.844733 21131 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7170713ab7ec4a419935ab4bfe2bdade [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:06:09.844857 21131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7170713ab7ec4a419935ab4bfe2bdade [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:09.846891 21131 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7170713ab7ec4a419935ab4bfe2bdade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7170713ab7ec4a419935ab4bfe2bdade" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37893 } }
I20260925 05:06:09.847297 21131 leader_election.cc:302] T 00000000000000000000000000000000 P 7170713ab7ec4a419935ab4bfe2bdade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7170713ab7ec4a419935ab4bfe2bdade; no voters:
I20260925 05:06:09.847870 21131 leader_election.cc:288] T 00000000000000000000000000000000 P 7170713ab7ec4a419935ab4bfe2bdade [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:06:09.847950 21141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7170713ab7ec4a419935ab4bfe2bdade [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:09.849891 21131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7170713ab7ec4a419935ab4bfe2bdade [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:06:09.850455 21141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7170713ab7ec4a419935ab4bfe2bdade [term 1 LEADER]: Becoming Leader. State: Replica: 7170713ab7ec4a419935ab4bfe2bdade, State: Running, Role: LEADER
I20260925 05:06:09.850996 21141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7170713ab7ec4a419935ab4bfe2bdade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7170713ab7ec4a419935ab4bfe2bdade" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37893 } }
I20260925 05:06:09.853615 21142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7170713ab7ec4a419935ab4bfe2bdade [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7170713ab7ec4a419935ab4bfe2bdade" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7170713ab7ec4a419935ab4bfe2bdade" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37893 } } }
I20260925 05:06:09.853735 21143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7170713ab7ec4a419935ab4bfe2bdade [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7170713ab7ec4a419935ab4bfe2bdade. Latest consensus state: current_term: 1 leader_uuid: "7170713ab7ec4a419935ab4bfe2bdade" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7170713ab7ec4a419935ab4bfe2bdade" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37893 } } }
I20260925 05:06:09.854012 21142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7170713ab7ec4a419935ab4bfe2bdade [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:09.854172 21143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7170713ab7ec4a419935ab4bfe2bdade [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:09.856035 21148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:06:09.861138 21148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:06:09.870069 21148 catalog_manager.cc:1347] Generated new cluster ID: 7a39b83ba0cb4b47bc68942ed77cf3e5
I20260925 05:06:09.870178 21148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:06:09.883747 21148 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:06:09.885582 21148 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:06:09.895183 21148 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7170713ab7ec4a419935ab4bfe2bdade: Generated new TSK 0
I20260925 05:06:09.896190 21148 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:06:09.903465 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 21064
W20260925 05:06:09.904747 21126 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56209
W20260925 05:06:09.905457 31708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 21064: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:06:09.905601 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 21064
W20260925 05:06:09.921927 21049 connection.cc:535] client connection to 127.30.247.62:37893 recv error: Network error: failed to read from TLS socket (remote: 127.30.247.62:37893): Connection reset by peer (error 104)
2026-09-25T05:06:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1843 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:06:10 dist-test-slave-5n86 krb5kdc[21162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:06:10 dist-test-slave-5n86 krb5kdc[21162](info): set up 2 sockets
Sep 25 05:06:10 dist-test-slave-5n86 krb5kdc[21162](info): commencing operation
krb5kdc: starting...
W20260925 05:06:11.005198 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:06:11 dist-test-slave-5n86 krb5kdc[21162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:06:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:06:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:06:11.248260 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:35353
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:37183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:35353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:11.567869 21175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:11.568147 21175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:11.568300 21175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:11.575407 21175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:11.575515 21175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:11.575598 21175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:11.575708 21175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:11.575785 21175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:11.585516 21175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:37183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:35353
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:35353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:11.586794 21175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:11.588138 21175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:11.599794 21183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:11.599859 21181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:11.599767 21180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:11.600827 21175 server_base.cc:1034] running on GCE node
I20260925 05:06:11.601627 21175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:11.602948 21175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:11.604146 21175 hybrid_clock.cc:648] HybridClock initialized: now 1790312771604091 us; error 44 us; skew 500 ppm
I20260925 05:06:11.604494 21175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:11 dist-test-slave-5n86 krb5kdc[21162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312771, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:11.613018 21175 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:06:11.614403 21175 webserver.cc:460] Webserver started at http://127.30.247.62:44307/ using document root <none> and password file <none>
I20260925 05:06:11.615036 21175 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:11.615134 21175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:11.615376 21175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:11.618261 21175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "902c2cc3f06a4e1096f0854cf3dc13d8"
format_stamp: "Formatted at 2026-09-25 05:06:11 on dist-test-slave-5n86"
server_key: "6640894d39f2deb14dc9e48b716e2e8b"
server_key_iv: "bab2b2b559f5e463843617e0a7fcffcd"
server_key_version: "encryptionkey@0"
I20260925 05:06:11.619010 21175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "902c2cc3f06a4e1096f0854cf3dc13d8"
format_stamp: "Formatted at 2026-09-25 05:06:11 on dist-test-slave-5n86"
server_key: "6640894d39f2deb14dc9e48b716e2e8b"
server_key_iv: "bab2b2b559f5e463843617e0a7fcffcd"
server_key_version: "encryptionkey@0"
I20260925 05:06:11.623929 21175 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 05:06:11.627360 21190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:11.628760 21175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:06:11.628890 21175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "902c2cc3f06a4e1096f0854cf3dc13d8"
format_stamp: "Formatted at 2026-09-25 05:06:11 on dist-test-slave-5n86"
server_key: "6640894d39f2deb14dc9e48b716e2e8b"
server_key_iv: "bab2b2b559f5e463843617e0a7fcffcd"
server_key_version: "encryptionkey@0"
I20260925 05:06:11.629053 21175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:11.634547 21175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:11.638435 21175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:11.638665 21175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:11.657346 21175 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:35353
I20260925 05:06:11.657385 21241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:35353 every 8 connection(s)
I20260925 05:06:11.658674 21175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:06:11.660202 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 21175
I20260925 05:06:11.660401 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:06:11.661013 31708 external_mini_cluster.cc:1428] Setting key 4c6aa36713d8f49b67e3cea15b4404a1
I20260925 05:06:11.663605 21242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:11.668620 21245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:11.662821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34855 (local address 127.30.247.62:35353)
0925 05:06:11.663573 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:11.663585 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:11.663627 (+ 42us) server_negotiation.cc:407] Connection header received
0925 05:06:11.665305 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:11.665326 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:11.665631 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:11.666016 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:11.667019 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:11.667025 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:11.667037 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:06:11.667050 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:11.667118 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:06:11.667140 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:11.667393 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:11.667548 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 05:06:11.667714 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":315,"threads_started":1}
I20260925 05:06:11.672013 31708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:06:11.675359 21242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902c2cc3f06a4e1096f0854cf3dc13d8: Bootstrap starting.
I20260925 05:06:11.678915 21245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:11.676240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36343 (local address 127.30.247.62:35353)
0925 05:06:11.676384 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:11.676387 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:11.677029 (+ 642us) server_negotiation.cc:407] Connection header received
0925 05:06:11.677203 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:11.677208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:11.677258 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:11.677367 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:11.678274 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:11.678280 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:11.678284 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:06:11.678306 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:11.678342 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:06:11.678356 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:11.678469 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:11.678606 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 05:06:11.678662 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 05:06:11.679065 21242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 902c2cc3f06a4e1096f0854cf3dc13d8: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:11.680250 21242 log.cc:824] T 00000000000000000000000000000000 P 902c2cc3f06a4e1096f0854cf3dc13d8: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:11.683099 21242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902c2cc3f06a4e1096f0854cf3dc13d8: No bootstrap required, opened a new log
I20260925 05:06:11.690929 21242 raft_consensus.cc:348] T 00000000000000000000000000000000 P 902c2cc3f06a4e1096f0854cf3dc13d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902c2cc3f06a4e1096f0854cf3dc13d8" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35353 } }
I20260925 05:06:11.691303 21242 raft_consensus.cc:374] T 00000000000000000000000000000000 P 902c2cc3f06a4e1096f0854cf3dc13d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:11.691433 21242 raft_consensus.cc:729] T 00000000000000000000000000000000 P 902c2cc3f06a4e1096f0854cf3dc13d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 902c2cc3f06a4e1096f0854cf3dc13d8, State: Initialized, Role: FOLLOWER
I20260925 05:06:11.691998 21242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 902c2cc3f06a4e1096f0854cf3dc13d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902c2cc3f06a4e1096f0854cf3dc13d8" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35353 } }
I20260925 05:06:11.692256 21242 raft_consensus.cc:388] T 00000000000000000000000000000000 P 902c2cc3f06a4e1096f0854cf3dc13d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:06:11.692408 21242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 902c2cc3f06a4e1096f0854cf3dc13d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:06:11.692571 21242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 902c2cc3f06a4e1096f0854cf3dc13d8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:11.694732 21242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 902c2cc3f06a4e1096f0854cf3dc13d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902c2cc3f06a4e1096f0854cf3dc13d8" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35353 } }
I20260925 05:06:11.695165 21242 leader_election.cc:302] T 00000000000000000000000000000000 P 902c2cc3f06a4e1096f0854cf3dc13d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 902c2cc3f06a4e1096f0854cf3dc13d8; no voters:
I20260925 05:06:11.695736 21242 leader_election.cc:288] T 00000000000000000000000000000000 P 902c2cc3f06a4e1096f0854cf3dc13d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:06:11.695796 21252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 902c2cc3f06a4e1096f0854cf3dc13d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:11.697063 21252 raft_consensus.cc:686] T 00000000000000000000000000000000 P 902c2cc3f06a4e1096f0854cf3dc13d8 [term 1 LEADER]: Becoming Leader. State: Replica: 902c2cc3f06a4e1096f0854cf3dc13d8, State: Running, Role: LEADER
I20260925 05:06:11.697558 21252 consensus_queue.cc:237] T 00000000000000000000000000000000 P 902c2cc3f06a4e1096f0854cf3dc13d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902c2cc3f06a4e1096f0854cf3dc13d8" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35353 } }
I20260925 05:06:11.697903 21242 sys_catalog.cc:564] T 00000000000000000000000000000000 P 902c2cc3f06a4e1096f0854cf3dc13d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:06:11.700268 21254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902c2cc3f06a4e1096f0854cf3dc13d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 902c2cc3f06a4e1096f0854cf3dc13d8. Latest consensus state: current_term: 1 leader_uuid: "902c2cc3f06a4e1096f0854cf3dc13d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902c2cc3f06a4e1096f0854cf3dc13d8" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35353 } } }
I20260925 05:06:11.700245 21253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902c2cc3f06a4e1096f0854cf3dc13d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "902c2cc3f06a4e1096f0854cf3dc13d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902c2cc3f06a4e1096f0854cf3dc13d8" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 35353 } } }
I20260925 05:06:11.700731 21253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902c2cc3f06a4e1096f0854cf3dc13d8 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:11.700731 21254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902c2cc3f06a4e1096f0854cf3dc13d8 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:11.702775 21260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:06:11.707803 21260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:06:11.716032 21260 catalog_manager.cc:1347] Generated new cluster ID: 977e78d53a4f41ec84a105f27a50ed38
I20260925 05:06:11.716126 21260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:06:11.745018 21260 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:06:11.745456 21260 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:06:11.753973 21260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 902c2cc3f06a4e1096f0854cf3dc13d8: Generated new TSK 0
I20260925 05:06:11.754729 21260 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:06:11.768119 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 21175
W20260925 05:06:11.769122 21237 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34855
W20260925 05:06:11.769743 31708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 21175: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:06:11.769894 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 21175
2026-09-25T05:06:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1864 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:06:11 dist-test-slave-5n86 krb5kdc[21273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:06:11 dist-test-slave-5n86 krb5kdc[21273](info): set up 2 sockets
Sep 25 05:06:11 dist-test-slave-5n86 krb5kdc[21273](info): commencing operation
krb5kdc: starting...
W20260925 05:06:12.885206 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.009s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:06:13 dist-test-slave-5n86 krb5kdc[21273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:06:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:06:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:06:13.119943 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:32873
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:42671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:32873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:13.447062 21286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:13.447311 21286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:13.447474 21286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:13.454355 21286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:13.454445 21286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:13.454555 21286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:13.454655 21286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:13.454733 21286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:13.463788 21286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:42671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:32873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:32873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:13.464932 21286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:13.466137 21286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:13.477463 21291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:13.477458 21294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:13.477494 21292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:13.478305 21286 server_base.cc:1034] running on GCE node
I20260925 05:06:13.478997 21286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:13.480281 21286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:13.481499 21286 hybrid_clock.cc:648] HybridClock initialized: now 1790312773481462 us; error 47 us; skew 500 ppm
I20260925 05:06:13.481825 21286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:13 dist-test-slave-5n86 krb5kdc[21273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312773, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:13.484826 21286 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:06:13.486135 21286 webserver.cc:460] Webserver started at http://127.30.247.62:45337/ using document root <none> and password file <none>
I20260925 05:06:13.486759 21286 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:13.486902 21286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:13.487186 21286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:13.490027 21286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "3676f9a6df7041f7b3a210032f2d1c58"
format_stamp: "Formatted at 2026-09-25 05:06:13 on dist-test-slave-5n86"
server_key: "f22c89224c3b593f97a7e1439513dd39"
server_key_iv: "73620671577791ed3a7e12426a3b3d3a"
server_key_version: "encryptionkey@0"
I20260925 05:06:13.490754 21286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "3676f9a6df7041f7b3a210032f2d1c58"
format_stamp: "Formatted at 2026-09-25 05:06:13 on dist-test-slave-5n86"
server_key: "f22c89224c3b593f97a7e1439513dd39"
server_key_iv: "73620671577791ed3a7e12426a3b3d3a"
server_key_version: "encryptionkey@0"
I20260925 05:06:13.495422 21286 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260925 05:06:13.499188 21301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:13.500425 21286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 05:06:13.500622 21286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "3676f9a6df7041f7b3a210032f2d1c58"
format_stamp: "Formatted at 2026-09-25 05:06:13 on dist-test-slave-5n86"
server_key: "f22c89224c3b593f97a7e1439513dd39"
server_key_iv: "73620671577791ed3a7e12426a3b3d3a"
server_key_version: "encryptionkey@0"
I20260925 05:06:13.500805 21286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:13.524015 21286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:13.528028 21286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:13.528287 21286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:13.545984 21286 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:32873
I20260925 05:06:13.546020 21352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:32873 every 8 connection(s)
I20260925 05:06:13.547174 21286 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:06:13.550545 21353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:13.550536 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 21286
I20260925 05:06:13.550737 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:06:13.551170 31708 external_mini_cluster.cc:1428] Setting key d806a30866117315bd8dcb69bf39f713
I20260925 05:06:13.564410 21353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3676f9a6df7041f7b3a210032f2d1c58: Bootstrap starting.
Sep 25 05:06:13 dist-test-slave-5n86 krb5kdc[21273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:06:13.567830 21353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3676f9a6df7041f7b3a210032f2d1c58: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:13.569196 21353 log.cc:824] T 00000000000000000000000000000000 P 3676f9a6df7041f7b3a210032f2d1c58: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:13.572292 21353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3676f9a6df7041f7b3a210032f2d1c58: No bootstrap required, opened a new log
I20260925 05:06:13.574370 21356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:13.552882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35167 (local address 127.30.247.62:32873)
0925 05:06:13.553746 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:13.553760 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:13.553816 (+ 56us) server_negotiation.cc:407] Connection header received
0925 05:06:13.555524 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:13.555549 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:13.556032 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:13.556444 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:13.557767 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:13.559712 (+ 1945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:13.561007 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:13.562060 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:13.562250 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:13.566090 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:13.566100 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:13.566117 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:13.566170 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:13.570060 (+ 3890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:13.570841 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:13.570848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:13.570855 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:13.571046 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:13.571387 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:13.571391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:13.571394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:13.572027 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:13.572336 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:13.572714 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 05:06:13.573136 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":374,"threads_started":1}
I20260925 05:06:13.578008 31708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:06:13.580906 21353 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3676f9a6df7041f7b3a210032f2d1c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3676f9a6df7041f7b3a210032f2d1c58" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 32873 } }
I20260925 05:06:13.581243 21353 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3676f9a6df7041f7b3a210032f2d1c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:13.581360 21353 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3676f9a6df7041f7b3a210032f2d1c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3676f9a6df7041f7b3a210032f2d1c58, State: Initialized, Role: FOLLOWER
I20260925 05:06:13.581961 21353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3676f9a6df7041f7b3a210032f2d1c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3676f9a6df7041f7b3a210032f2d1c58" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 32873 } }
I20260925 05:06:13.582294 21353 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3676f9a6df7041f7b3a210032f2d1c58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:06:13.582439 21353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3676f9a6df7041f7b3a210032f2d1c58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:06:13.582643 21353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3676f9a6df7041f7b3a210032f2d1c58 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:13.585047 21353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3676f9a6df7041f7b3a210032f2d1c58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3676f9a6df7041f7b3a210032f2d1c58" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 32873 } }
I20260925 05:06:13.585562 21353 leader_election.cc:302] T 00000000000000000000000000000000 P 3676f9a6df7041f7b3a210032f2d1c58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3676f9a6df7041f7b3a210032f2d1c58; no voters:
I20260925 05:06:13.586223 21353 leader_election.cc:288] T 00000000000000000000000000000000 P 3676f9a6df7041f7b3a210032f2d1c58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:06:13.586309 21363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3676f9a6df7041f7b3a210032f2d1c58 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:13.587561 21363 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3676f9a6df7041f7b3a210032f2d1c58 [term 1 LEADER]: Becoming Leader. State: Replica: 3676f9a6df7041f7b3a210032f2d1c58, State: Running, Role: LEADER
I20260925 05:06:13.588138 21363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3676f9a6df7041f7b3a210032f2d1c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3676f9a6df7041f7b3a210032f2d1c58" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 32873 } }
I20260925 05:06:13.588440 21353 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3676f9a6df7041f7b3a210032f2d1c58 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:06:13.596264 21365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3676f9a6df7041f7b3a210032f2d1c58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3676f9a6df7041f7b3a210032f2d1c58. Latest consensus state: current_term: 1 leader_uuid: "3676f9a6df7041f7b3a210032f2d1c58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3676f9a6df7041f7b3a210032f2d1c58" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 32873 } } }
I20260925 05:06:13.596750 21365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3676f9a6df7041f7b3a210032f2d1c58 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:13.597376 21364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3676f9a6df7041f7b3a210032f2d1c58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3676f9a6df7041f7b3a210032f2d1c58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3676f9a6df7041f7b3a210032f2d1c58" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 32873 } } }
I20260925 05:06:13.597645 21364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3676f9a6df7041f7b3a210032f2d1c58 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:13.597718 21356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:13.582198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32801 (local address 127.30.247.62:32873)
0925 05:06:13.582371 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:13.582377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:13.582998 (+ 621us) server_negotiation.cc:407] Connection header received
0925 05:06:13.583098 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:13.583103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:13.583173 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:13.583284 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:13.584663 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:13.585713 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:13.587281 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:13.587934 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:13.588081 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:13.590897 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:13.590902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:13.590907 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:13.590939 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:13.593670 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:13.594416 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:13.594424 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:13.594430 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:13.594562 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:13.595033 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:13.595041 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:13.595047 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:13.595536 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:13.595819 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:13.596875 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0925 05:06:13.597108 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 05:06:13.597671 21373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:06:13.604449 21373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:06:13.613724 21373 catalog_manager.cc:1347] Generated new cluster ID: fb4b25ae62a34729999e31d4f4873da3
I20260925 05:06:13.613839 21373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:06:13.623611 21373 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:06:13.624821 21373 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:06:13.630582 21373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3676f9a6df7041f7b3a210032f2d1c58: Generated new TSK 0
I20260925 05:06:13.631242 21373 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:06:13.642303 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 21286
I20260925 05:06:13.643337 21348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:06:13.730459 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 21286
2026-09-25T05:06:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1957 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:06:13 dist-test-slave-5n86 krb5kdc[21386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:06:13 dist-test-slave-5n86 krb5kdc[21386](info): set up 2 sockets
Sep 25 05:06:13 dist-test-slave-5n86 krb5kdc[21386](info): commencing operation
krb5kdc: starting...
W20260925 05:06:14.817262 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.004s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:06:14 dist-test-slave-5n86 krb5kdc[21386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:06:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:06:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:06:15.060317 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:37767
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:37037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:37767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:15.382202 21399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:15.382494 21399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:15.382645 21399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:15.389513 21399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:15.389603 21399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:15.389719 21399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:15.389819 21399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:15.389881 21399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:15.399142 21399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:37037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:37767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:37767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:15.400344 21399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:15.401688 21399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:15.413130 21404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:15.413208 21405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:15.413175 21407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:15.414036 21399 server_base.cc:1034] running on GCE node
I20260925 05:06:15.414697 21399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:15.415975 21399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:15.417167 21399 hybrid_clock.cc:648] HybridClock initialized: now 1790312775417138 us; error 29 us; skew 500 ppm
I20260925 05:06:15.417491 21399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:15 dist-test-slave-5n86 krb5kdc[21386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312775, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:15.420511 21399 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:06:15.421911 21399 webserver.cc:460] Webserver started at http://127.30.247.62:39201/ using document root <none> and password file <none>
I20260925 05:06:15.422505 21399 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:15.422600 21399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:15.422833 21399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:15.425637 21399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "17464ccfcf8c40988f4738e3752947f4"
format_stamp: "Formatted at 2026-09-25 05:06:15 on dist-test-slave-5n86"
server_key: "b243e16678b9d0de917fc9d7db7ce6b3"
server_key_iv: "f9cd33cf7a527289a0bf3ecf1eb655d5"
server_key_version: "encryptionkey@0"
I20260925 05:06:15.426308 21399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "17464ccfcf8c40988f4738e3752947f4"
format_stamp: "Formatted at 2026-09-25 05:06:15 on dist-test-slave-5n86"
server_key: "b243e16678b9d0de917fc9d7db7ce6b3"
server_key_iv: "f9cd33cf7a527289a0bf3ecf1eb655d5"
server_key_version: "encryptionkey@0"
I20260925 05:06:15.431033 21399 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 05:06:15.434154 21414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:15.435345 21399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 05:06:15.435523 21399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "17464ccfcf8c40988f4738e3752947f4"
format_stamp: "Formatted at 2026-09-25 05:06:15 on dist-test-slave-5n86"
server_key: "b243e16678b9d0de917fc9d7db7ce6b3"
server_key_iv: "f9cd33cf7a527289a0bf3ecf1eb655d5"
server_key_version: "encryptionkey@0"
I20260925 05:06:15.435709 21399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:15.475242 21399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:15.479404 21399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:15.479681 21399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:15.498175 21399 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:37767
I20260925 05:06:15.498209 21465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:37767 every 8 connection(s)
I20260925 05:06:15.499470 21399 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:06:15.503188 21466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:15.503284 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 21399
I20260925 05:06:15.503451 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:06:15.503993 31708 external_mini_cluster.cc:1428] Setting key 9869cb4c5293faf4bb55e3fdf156cc99
I20260925 05:06:15.516800 21466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17464ccfcf8c40988f4738e3752947f4: Bootstrap starting.
I20260925 05:06:15.517392 21469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:15.505749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60657 (local address 127.30.247.62:37767)
0925 05:06:15.506576 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:15.506587 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:15.506631 (+ 44us) server_negotiation.cc:407] Connection header received
0925 05:06:15.508122 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:15.508143 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:15.508516 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:15.508921 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:15.509533 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:15.510893 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:15.512768 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:15.513452 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:15.513594 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:15.514865 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:15.514874 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:15.514891 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:06:15.515492 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:15.515571 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:06:15.515604 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:15.515709 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:15.515950 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 05:06:15.516231 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":403,"threads_started":1}
I20260925 05:06:15.519836 21466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17464ccfcf8c40988f4738e3752947f4: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:15.521121 31708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:06:15.521093 21466 log.cc:824] T 00000000000000000000000000000000 P 17464ccfcf8c40988f4738e3752947f4: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:15.524154 21466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17464ccfcf8c40988f4738e3752947f4: No bootstrap required, opened a new log
I20260925 05:06:15.532518 21469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:15.526097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45661 (local address 127.30.247.62:37767)
0925 05:06:15.526286 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:15.526293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:15.526703 (+ 410us) server_negotiation.cc:407] Connection header received
0925 05:06:15.526843 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:15.526856 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:15.526942 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:15.527047 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:15.527504 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:15.528626 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:15.529898 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:15.530601 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:15.530727 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:15.531727 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:15.531734 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:15.531738 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:06:15.531759 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:15.531807 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:06:15.531827 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:15.531941 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:15.532052 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 05:06:15.532159 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 05:06:15.533591 21466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17464ccfcf8c40988f4738e3752947f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17464ccfcf8c40988f4738e3752947f4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37767 } }
I20260925 05:06:15.533947 21466 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17464ccfcf8c40988f4738e3752947f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:15.534053 21466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17464ccfcf8c40988f4738e3752947f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17464ccfcf8c40988f4738e3752947f4, State: Initialized, Role: FOLLOWER
I20260925 05:06:15.534644 21466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17464ccfcf8c40988f4738e3752947f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17464ccfcf8c40988f4738e3752947f4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37767 } }
I20260925 05:06:15.534878 21466 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17464ccfcf8c40988f4738e3752947f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:06:15.535020 21466 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17464ccfcf8c40988f4738e3752947f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:06:15.535199 21466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17464ccfcf8c40988f4738e3752947f4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:15.537336 21466 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17464ccfcf8c40988f4738e3752947f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17464ccfcf8c40988f4738e3752947f4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37767 } }
I20260925 05:06:15.537814 21466 leader_election.cc:302] T 00000000000000000000000000000000 P 17464ccfcf8c40988f4738e3752947f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17464ccfcf8c40988f4738e3752947f4; no voters:
I20260925 05:06:15.538393 21466 leader_election.cc:288] T 00000000000000000000000000000000 P 17464ccfcf8c40988f4738e3752947f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:06:15.538497 21476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17464ccfcf8c40988f4738e3752947f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:15.539458 21476 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17464ccfcf8c40988f4738e3752947f4 [term 1 LEADER]: Becoming Leader. State: Replica: 17464ccfcf8c40988f4738e3752947f4, State: Running, Role: LEADER
I20260925 05:06:15.539932 21476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17464ccfcf8c40988f4738e3752947f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17464ccfcf8c40988f4738e3752947f4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37767 } }
I20260925 05:06:15.540589 21466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17464ccfcf8c40988f4738e3752947f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:06:15.542928 21478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17464ccfcf8c40988f4738e3752947f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17464ccfcf8c40988f4738e3752947f4. Latest consensus state: current_term: 1 leader_uuid: "17464ccfcf8c40988f4738e3752947f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17464ccfcf8c40988f4738e3752947f4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37767 } } }
I20260925 05:06:15.543035 21477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17464ccfcf8c40988f4738e3752947f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17464ccfcf8c40988f4738e3752947f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17464ccfcf8c40988f4738e3752947f4" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 37767 } } }
I20260925 05:06:15.543336 21477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17464ccfcf8c40988f4738e3752947f4 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:15.543336 21478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17464ccfcf8c40988f4738e3752947f4 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:15.548496 21484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:06:15.552436 21484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:06:15.560932 21484 catalog_manager.cc:1347] Generated new cluster ID: 34aa1b82290b4006a93b6623ca6310cc
I20260925 05:06:15.561059 21484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:06:15.577778 21484 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:06:15.579458 21484 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:06:15.585583 21484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17464ccfcf8c40988f4738e3752947f4: Generated new TSK 0
I20260925 05:06:15.586320 21484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:06:15.594661 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 21399
W20260925 05:06:15.595651 21461 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60657
W20260925 05:06:15.596306 31708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 21399: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:06:15.596474 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 21399
2026-09-25T05:06:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1865 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:06:15 dist-test-slave-5n86 krb5kdc[21497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:06:15 dist-test-slave-5n86 krb5kdc[21497](info): set up 2 sockets
Sep 25 05:06:15 dist-test-slave-5n86 krb5kdc[21497](info): commencing operation
krb5kdc: starting...
W20260925 05:06:16.701284 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:06:16 dist-test-slave-5n86 krb5kdc[21497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:06:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:06:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:06:16.948210 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:40777
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:40777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:17.264999 21510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:17.265273 21510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:17.265465 21510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:17.272471 21510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:17.272640 21510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:17.272714 21510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:17.272778 21510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:17.272822 21510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:17.282260 21510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:40777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:40777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:17.283406 21510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:17.284655 21510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:17.296039 21516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:17.296962 21515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:17.297256 21518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:17.298200 21510 server_base.cc:1034] running on GCE node
I20260925 05:06:17.298947 21510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:17.300318 21510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:17.301566 21510 hybrid_clock.cc:648] HybridClock initialized: now 1790312777301528 us; error 49 us; skew 500 ppm
I20260925 05:06:17.301916 21510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:17 dist-test-slave-5n86 krb5kdc[21497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312777, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:17.308604 21510 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:06:17.310078 21510 webserver.cc:460] Webserver started at http://127.30.247.62:40879/ using document root <none> and password file <none>
I20260925 05:06:17.310777 21510 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:17.310932 21510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:17.311245 21510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:17.314201 21510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "7d8b9e1fb8f84ed591c7ec8b0100650b"
format_stamp: "Formatted at 2026-09-25 05:06:17 on dist-test-slave-5n86"
server_key: "71b38a58889d3ef8465715baca0acdcf"
server_key_iv: "afcc52a708b9c1b72a01a5899b0c8e83"
server_key_version: "encryptionkey@0"
I20260925 05:06:17.314973 21510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "7d8b9e1fb8f84ed591c7ec8b0100650b"
format_stamp: "Formatted at 2026-09-25 05:06:17 on dist-test-slave-5n86"
server_key: "71b38a58889d3ef8465715baca0acdcf"
server_key_iv: "afcc52a708b9c1b72a01a5899b0c8e83"
server_key_version: "encryptionkey@0"
I20260925 05:06:17.319962 21510 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 05:06:17.323508 21525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:17.324928 21510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:06:17.325127 21510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "7d8b9e1fb8f84ed591c7ec8b0100650b"
format_stamp: "Formatted at 2026-09-25 05:06:17 on dist-test-slave-5n86"
server_key: "71b38a58889d3ef8465715baca0acdcf"
server_key_iv: "afcc52a708b9c1b72a01a5899b0c8e83"
server_key_version: "encryptionkey@0"
I20260925 05:06:17.325325 21510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:17.331065 21510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:17.335302 21510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:17.335603 21510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:17.353771 21510 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:40777
I20260925 05:06:17.353816 21576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:40777 every 8 connection(s)
I20260925 05:06:17.355098 21510 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:06:17.358541 21577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:17.359409 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 21510
I20260925 05:06:17.359592 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:06:17.360133 31708 external_mini_cluster.cc:1428] Setting key 5b99a072a2b714d26c7d3f90e020e7e5
I20260925 05:06:17.368023 21580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:17.361994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46189 (local address 127.30.247.62:40777)
0925 05:06:17.362794 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:17.362806 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:17.362845 (+ 39us) server_negotiation.cc:407] Connection header received
0925 05:06:17.364632 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:17.364663 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:17.364993 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:17.365314 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:17.366444 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:17.366450 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:17.366463 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:06:17.366487 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:17.366553 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:06:17.366573 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:17.366740 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:17.366890 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 05:06:17.367125 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":387,"threads_started":1}
I20260925 05:06:17.371389 21577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d8b9e1fb8f84ed591c7ec8b0100650b: Bootstrap starting.
I20260925 05:06:17.371658 31708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:06:17.375324 21577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d8b9e1fb8f84ed591c7ec8b0100650b: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:17.376662 21577 log.cc:824] T 00000000000000000000000000000000 P 7d8b9e1fb8f84ed591c7ec8b0100650b: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:17.379849 21577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d8b9e1fb8f84ed591c7ec8b0100650b: No bootstrap required, opened a new log
I20260925 05:06:17.380141 21580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:17.377083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51593 (local address 127.30.247.62:40777)
0925 05:06:17.377299 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:17.377311 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:17.378330 (+ 1019us) server_negotiation.cc:407] Connection header received
0925 05:06:17.378481 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:17.378484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:17.378547 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:17.378624 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:17.379517 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:17.379536 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:17.379539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:06:17.379553 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:17.379588 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:06:17.379602 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:17.379684 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:17.379832 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 05:06:17.379877 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 05:06:17.387269 21577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d8b9e1fb8f84ed591c7ec8b0100650b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8b9e1fb8f84ed591c7ec8b0100650b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40777 } }
I20260925 05:06:17.387645 21577 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d8b9e1fb8f84ed591c7ec8b0100650b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:17.387753 21577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d8b9e1fb8f84ed591c7ec8b0100650b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d8b9e1fb8f84ed591c7ec8b0100650b, State: Initialized, Role: FOLLOWER
I20260925 05:06:17.388235 21577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d8b9e1fb8f84ed591c7ec8b0100650b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8b9e1fb8f84ed591c7ec8b0100650b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40777 } }
I20260925 05:06:17.388455 21577 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d8b9e1fb8f84ed591c7ec8b0100650b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:06:17.388616 21577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d8b9e1fb8f84ed591c7ec8b0100650b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:06:17.388746 21577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d8b9e1fb8f84ed591c7ec8b0100650b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:17.391091 21577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d8b9e1fb8f84ed591c7ec8b0100650b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8b9e1fb8f84ed591c7ec8b0100650b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40777 } }
I20260925 05:06:17.391505 21577 leader_election.cc:302] T 00000000000000000000000000000000 P 7d8b9e1fb8f84ed591c7ec8b0100650b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d8b9e1fb8f84ed591c7ec8b0100650b; no voters:
I20260925 05:06:17.391985 21577 leader_election.cc:288] T 00000000000000000000000000000000 P 7d8b9e1fb8f84ed591c7ec8b0100650b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:06:17.392110 21587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d8b9e1fb8f84ed591c7ec8b0100650b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:17.392866 21587 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d8b9e1fb8f84ed591c7ec8b0100650b [term 1 LEADER]: Becoming Leader. State: Replica: 7d8b9e1fb8f84ed591c7ec8b0100650b, State: Running, Role: LEADER
I20260925 05:06:17.393396 21587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d8b9e1fb8f84ed591c7ec8b0100650b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8b9e1fb8f84ed591c7ec8b0100650b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40777 } }
I20260925 05:06:17.394202 21577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d8b9e1fb8f84ed591c7ec8b0100650b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:06:17.396188 21589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d8b9e1fb8f84ed591c7ec8b0100650b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d8b9e1fb8f84ed591c7ec8b0100650b. Latest consensus state: current_term: 1 leader_uuid: "7d8b9e1fb8f84ed591c7ec8b0100650b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8b9e1fb8f84ed591c7ec8b0100650b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40777 } } }
I20260925 05:06:17.396699 21589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d8b9e1fb8f84ed591c7ec8b0100650b [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:17.397403 21588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d8b9e1fb8f84ed591c7ec8b0100650b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d8b9e1fb8f84ed591c7ec8b0100650b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8b9e1fb8f84ed591c7ec8b0100650b" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 40777 } } }
I20260925 05:06:17.397750 21588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d8b9e1fb8f84ed591c7ec8b0100650b [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:17.399258 21593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:06:17.404626 21593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:06:17.413146 21593 catalog_manager.cc:1347] Generated new cluster ID: 03621d50007541c28ed2c6d3edd8212b
I20260925 05:06:17.413306 21593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:06:17.428049 21593 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:06:17.428488 21593 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:06:17.438481 21593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d8b9e1fb8f84ed591c7ec8b0100650b: Generated new TSK 0
I20260925 05:06:17.439145 21593 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:06:17.449345 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 21510
W20260925 05:06:17.450429 21572 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46189
W20260925 05:06:17.451282 31708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 21510: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:06:17.451504 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 21510
2026-09-25T05:06:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1859 ms)
[----------] 18 tests from AuthTokenIssuingTest (34209 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:06:17 dist-test-slave-5n86 krb5kdc[21608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:06:17 dist-test-slave-5n86 krb5kdc[21608](info): set up 2 sockets
Sep 25 05:06:17 dist-test-slave-5n86 krb5kdc[21608](info): commencing operation
krb5kdc: starting...
W20260925 05:06:18.549324 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:06:18 dist-test-slave-5n86 krb5kdc[21608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:06:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:06:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:06:18.792433 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:46773
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:46773,127.30.247.61:40489,127.30.247.60:43793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:19.115278 21621 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:06:19.115578 21621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:19.115661 21621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:19.115743 21621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:19.122689 21621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:19.122773 21621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:19.122828 21621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:19.122882 21621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:19.122941 21621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:19.132365 21621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:46773,127.30.247.61:40489,127.30.247.60:43793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:46773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:19.133647 21621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:19.134836 21621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:19.146468 21626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:19.146396 21627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:19.146436 21629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:19.147388 21621 server_base.cc:1034] running on GCE node
I20260925 05:06:19.148093 21621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:19.149426 21621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:19.150629 21621 hybrid_clock.cc:648] HybridClock initialized: now 1790312779150578 us; error 25 us; skew 500 ppm
I20260925 05:06:19.150950 21621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:19 dist-test-slave-5n86 krb5kdc[21608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312779, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:19.154055 21621 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:06:19.155429 21621 webserver.cc:460] Webserver started at http://127.30.247.62:33809/ using document root <none> and password file <none>
I20260925 05:06:19.156013 21621 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:19.156090 21621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:19.156343 21621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:19.159140 21621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "ce8b4a6e862f404693ff65c3c3c56353"
format_stamp: "Formatted at 2026-09-25 05:06:19 on dist-test-slave-5n86"
server_key: "ce6a893c1baabd7d088b1967b99f811f"
server_key_iv: "f2bb8b03f3529e024fce8efab0853efd"
server_key_version: "encryptionkey@0"
I20260925 05:06:19.159837 21621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "ce8b4a6e862f404693ff65c3c3c56353"
format_stamp: "Formatted at 2026-09-25 05:06:19 on dist-test-slave-5n86"
server_key: "ce6a893c1baabd7d088b1967b99f811f"
server_key_iv: "f2bb8b03f3529e024fce8efab0853efd"
server_key_version: "encryptionkey@0"
I20260925 05:06:19.164772 21621 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 05:06:19.168004 21636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:19.169364 21621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:06:19.169535 21621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "ce8b4a6e862f404693ff65c3c3c56353"
format_stamp: "Formatted at 2026-09-25 05:06:19 on dist-test-slave-5n86"
server_key: "ce6a893c1baabd7d088b1967b99f811f"
server_key_iv: "f2bb8b03f3529e024fce8efab0853efd"
server_key_version: "encryptionkey@0"
I20260925 05:06:19.169713 21621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:19.186905 21621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:19.191159 21621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:19.191422 21621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:19.209048 21621 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:46773
I20260925 05:06:19.209071 21687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:46773 every 8 connection(s)
I20260925 05:06:19.210316 21621 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:06:19.213810 21688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:19.214821 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 21621
I20260925 05:06:19.215059 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:06:19.215710 31708 external_mini_cluster.cc:1428] Setting key e440a3163180975722a1334d93b5ab35
I20260925 05:06:19.220237 21688 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46773 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:06:19 dist-test-slave-5n86 krb5kdc[21608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312779, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.247.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.61@KRBTEST.COM" created.
I20260925 05:06:19.244028 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.223237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37816 (local address 127.30.247.62:46773)
0925 05:06:19.225081 (+ 1844us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:19.225091 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:19.225147 (+ 56us) server_negotiation.cc:407] Connection header received
0925 05:06:19.226586 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:19.226596 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:19.226919 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:19.227130 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:19.228249 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.230180 (+ 1931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.231547 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.232510 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.232737 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:19.235923 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:19.235935 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:19.235953 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:19.236001 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:19.239902 (+ 3901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:19.240619 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:19.240635 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:19.240639 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:19.240841 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:19.241308 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:19.241312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:19.241314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:19.241824 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:19.242091 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:19.242610 (+ 519us) server_negotiation.cc:299] Negotiation successful
0925 05:06:19.242936 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":118,"server-negotiator.queue_time_us":1608,"spinlock_wait_cycles":51968,"thread_start_us":1419,"threads_started":1}
I20260925 05:06:19.244076 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.223238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:46773 (local address 127.0.0.1:37816)
0925 05:06:19.224191 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.224262 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:19.226222 (+ 1960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:19.227160 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:19.227165 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:19.227698 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:19.228025 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:19.228050 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.230322 (+ 2272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.230325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:19.231403 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:19.231414 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.232738 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.232743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:19.232913 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:19.233583 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:19.233604 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:19.235695 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:19.240059 (+ 4364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:19.240068 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:19.240077 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:19.240457 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:19.240978 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:19.240983 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:19.240988 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:19.241187 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:19.242035 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:19.242041 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:19.242234 (+ 193us) client_negotiation.cc:769] Sending connection context
0925 05:06:19.242441 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 05:06:19.242812 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":42,"thread_start_us":388,"threads_started":1}
I20260925 05:06:19.246922 21688 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.61" port: 40489 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:06:19.247969 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.247568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:40489 (local address 127.0.0.1:38300)
0925 05:06:19.247743 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.247858 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260925 05:06:19.248291 21637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.247.61:40489: connect: Connection refused (error 111)
I20260925 05:06:19.249817 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.249482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:40489 (local address 127.0.0.1:38310)
0925 05:06:19.249637 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.249702 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 05:06:19.250103 21688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.61:40489: Network error: Client connection negotiation failed: client connection to 127.30.247.61:40489: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.61.keytab.
Entry for principal kudu/127.30.247.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.61.keytab.
I20260925 05:06:19.276438 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.61:40489
--webserver_interface=127.30.247.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.61.keytab
--principal=kudu/127.30.247.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:46773,127.30.247.61:40489,127.30.247.60:43793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:06:19.299463 21688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.61" port: 40489 } attempt: 1
I20260925 05:06:19.300738 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.300227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:40489 (local address 127.0.0.1:38318)
0925 05:06:19.300428 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.300529 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 05:06:19.301848 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.301424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:40489 (local address 127.0.0.1:38324)
0925 05:06:19.301609 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.301696 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260925 05:06:19.302079 21688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.61:40489: Network error: Client connection negotiation failed: client connection to 127.30.247.61:40489: connect: Connection refused (error 111)
I20260925 05:06:19.370360 21688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.61" port: 40489 } attempt: 2
I20260925 05:06:19.371646 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.371079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:40489 (local address 127.0.0.1:43758)
0925 05:06:19.371354 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.371471 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 05:06:19.372785 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.372322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:40489 (local address 127.0.0.1:43770)
0925 05:06:19.372509 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.372631 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 05:06:19.373087 21688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.61:40489: Network error: Client connection negotiation failed: client connection to 127.30.247.61:40489: connect: Connection refused (error 111)
I20260925 05:06:19.464417 21688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.61" port: 40489 } attempt: 3
I20260925 05:06:19.465824 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.465214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:40489 (local address 127.0.0.1:43774)
0925 05:06:19.465487 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.465624 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 05:06:19.466950 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.466590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:40489 (local address 127.0.0.1:43782)
0925 05:06:19.466743 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.466816 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 05:06:19.467288 21688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.61:40489: Network error: Client connection negotiation failed: client connection to 127.30.247.61:40489: connect: Connection refused (error 111)
W20260925 05:06:19.600903 21695 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:06:19.601253 21695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:19.601375 21695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:19.601533 21695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:19.608523 21695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:19.608620 21695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:19.608739 21695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:19.608795 21695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:19.608852 21695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:19.610548 21688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.61" port: 40489 } attempt: 4
I20260925 05:06:19.611790 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.611265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:40489 (local address 127.0.0.1:43792)
0925 05:06:19.611562 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.611652 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260925 05:06:19.612772 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.612419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:40489 (local address 127.0.0.1:43806)
0925 05:06:19.612583 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.612663 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 05:06:19.612972 21688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.61:40489: Network error: Client connection negotiation failed: client connection to 127.30.247.61:40489: connect: Connection refused (error 111)
I20260925 05:06:19.618857 21695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:46773,127.30.247.61:40489,127.30.247.60:43793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.61.keytab
--principal=kudu/127.30.247.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.61:40489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.247.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:19.620097 21695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:19.621446 21695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:19.632961 21701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:19.632968 21703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:19.633302 21700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:19.633716 21695 server_base.cc:1034] running on GCE node
I20260925 05:06:19.634536 21695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:19.635823 21695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:19.637048 21695 hybrid_clock.cc:648] HybridClock initialized: now 1790312779636957 us; error 58 us; skew 500 ppm
I20260925 05:06:19.637380 21695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:19 dist-test-slave-5n86 krb5kdc[21608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312779, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:19.640455 21695 init.cc:377] Logged in from keytab as kudu/127.30.247.61@KRBTEST.COM (short username kudu)
I20260925 05:06:19.641881 21695 webserver.cc:460] Webserver started at http://127.30.247.61:36723/ using document root <none> and password file <none>
I20260925 05:06:19.642561 21695 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:19.642675 21695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:19.643002 21695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:19.645910 21695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-1/data/instance:
uuid: "d85ddd55f67447ad8abc89c1c14c4412"
format_stamp: "Formatted at 2026-09-25 05:06:19 on dist-test-slave-5n86"
server_key: "36a0bc01b6f589f70f2cdb2c59d407a0"
server_key_iv: "44dc2ec3d40e836b718470e6a39e3ffc"
server_key_version: "encryptionkey@0"
I20260925 05:06:19.646705 21695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-1/wal/instance:
uuid: "d85ddd55f67447ad8abc89c1c14c4412"
format_stamp: "Formatted at 2026-09-25 05:06:19 on dist-test-slave-5n86"
server_key: "36a0bc01b6f589f70f2cdb2c59d407a0"
server_key_iv: "44dc2ec3d40e836b718470e6a39e3ffc"
server_key_version: "encryptionkey@0"
I20260925 05:06:19.651729 21695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 05:06:19.655000 21710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:19.656309 21695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 05:06:19.656494 21695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-1/wal
uuid: "d85ddd55f67447ad8abc89c1c14c4412"
format_stamp: "Formatted at 2026-09-25 05:06:19 on dist-test-slave-5n86"
server_key: "36a0bc01b6f589f70f2cdb2c59d407a0"
server_key_iv: "44dc2ec3d40e836b718470e6a39e3ffc"
server_key_version: "encryptionkey@0"
I20260925 05:06:19.656769 21695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:19.674242 21695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:19.678295 21695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:19.678566 21695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:19.697031 21695 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.61:40489
I20260925 05:06:19.697044 21761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.61:40489 every 8 connection(s)
I20260925 05:06:19.698318 21695 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-1/data/info.pb
I20260925 05:06:19.699744 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 21695
I20260925 05:06:19.699934 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-1/wal/instance
I20260925 05:06:19.700475 31708 external_mini_cluster.cc:1428] Setting key 1c8a962b9cdfa3dd2506f10673fe2d8a
I20260925 05:06:19.702734 21762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:19.712190 21762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46773 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:06:19 dist-test-slave-5n86 krb5kdc[21608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312779, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.61@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.247.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.60@KRBTEST.COM" created.
I20260925 05:06:19.733398 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.714879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55474 (local address 127.30.247.62:46773)
0925 05:06:19.715133 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:19.715142 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:19.716190 (+ 1048us) server_negotiation.cc:407] Connection header received
0925 05:06:19.718085 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:19.718094 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:19.718186 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:19.718307 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:19.719722 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.720907 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.722487 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.723246 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.723400 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:19.727197 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:19.727202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:19.727205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:19.727238 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:19.730178 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:19.731118 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:19.731122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:19.731124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:19.731238 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:19.731732 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:19.731735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:19.731736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:19.732005 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:19.732122 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:19.733011 (+ 889us) server_negotiation.cc:299] Negotiation successful
0925 05:06:19.733174 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 05:06:19.734937 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.715210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:46773 (local address 127.0.0.1:55474)
0925 05:06:19.716008 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.716061 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:19.717818 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:19.718489 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:19.718496 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:19.719000 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:19.719527 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:19.719543 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.721056 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.721061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:19.722314 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:19.722332 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.723507 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.723513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:19.723714 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:19.724447 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:19.724470 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:19.726984 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:19.730329 (+ 3345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:19.730340 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:19.730360 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:19.731000 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:19.731348 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:19.731355 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:19.731361 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:19.731620 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:19.732135 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:19.732145 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:19.732624 (+ 479us) client_negotiation.cc:769] Sending connection context
0925 05:06:19.732919 (+ 295us) client_negotiation.cc:240] Negotiation successful
0925 05:06:19.733430 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":347,"threads_started":1}
I20260925 05:06:19.736922 21762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.61" port: 40489 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:06:19 dist-test-slave-5n86 krb5kdc[21608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312779, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.61@KRBTEST.COM for kudu/127.30.247.61@KRBTEST.COM
I20260925 05:06:19.750727 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.737644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:40489 (local address 127.0.0.1:43812)
0925 05:06:19.737879 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.737894 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:19.737984 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:19.738954 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:19.738958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:19.739372 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:19.739488 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:19.739496 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.740895 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.740903 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:19.741735 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:19.741745 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.742553 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.742558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:19.742754 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:19.743325 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:19.743341 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:19.745302 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:19.748300 (+ 2998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:19.748308 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:19.748312 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:19.748733 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:19.749093 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:19.749099 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:19.749103 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:19.749240 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:19.749800 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:19.749807 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:19.749952 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 05:06:19.750099 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 05:06:19.750355 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":7}
I20260925 05:06:19.751068 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.737934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43812 (local address 127.30.247.61:40489)
0925 05:06:19.738552 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:19.738558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:19.738581 (+ 23us) server_negotiation.cc:407] Connection header received
0925 05:06:19.738620 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:19.738625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:19.738783 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:19.738950 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:19.739661 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.740763 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.741875 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.742380 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.742543 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:19.745510 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:19.745521 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:19.745533 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:19.745562 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:19.748167 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:19.748856 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:19.748866 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:19.748873 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:19.748981 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:19.749343 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:19.749350 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:19.749353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:19.749673 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:19.749896 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:19.750415 (+ 519us) server_negotiation.cc:299] Negotiation successful
0925 05:06:19.750642 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":483,"thread_start_us":340,"threads_started":1}
I20260925 05:06:19.753258 21762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:06:19.754431 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.754037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:43793 (local address 127.0.0.1:58356)
0925 05:06:19.754194 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.754298 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 05:06:19.754730 21712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
I20260925 05:06:19.756227 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.755892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:43793 (local address 127.0.0.1:58360)
0925 05:06:19.756055 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.756135 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260925 05:06:19.756433 21762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.60:43793: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.60.keytab.
Entry for principal kudu/127.30.247.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.60.keytab.
I20260925 05:06:19.763839 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.60:43793
--webserver_interface=127.30.247.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.60.keytab
--principal=kudu/127.30.247.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:46773,127.30.247.61:40489,127.30.247.60:43793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:06:19.805823 21762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } attempt: 1
I20260925 05:06:19.807108 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.806532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:43793 (local address 127.0.0.1:58366)
0925 05:06:19.806763 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.806900 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 05:06:19.808332 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.807844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:43793 (local address 127.0.0.1:58370)
0925 05:06:19.808056 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.808167 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 05:06:19.808621 21762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.60:43793: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
I20260925 05:06:19.876869 21762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } attempt: 2
I20260925 05:06:19.878105 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.877570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:43793 (local address 127.0.0.1:58386)
0925 05:06:19.877795 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.877906 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 05:06:19.879153 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.878781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:43793 (local address 127.0.0.1:58402)
0925 05:06:19.878928 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.879018 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 05:06:19.879369 21762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.60:43793: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
I20260925 05:06:19.912274 21688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.61" port: 40489 } attempt: 5
Sep 25 05:06:19 dist-test-slave-5n86 krb5kdc[21608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312779, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for kudu/127.30.247.61@KRBTEST.COM
I20260925 05:06:19.925624 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.913238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43818 (local address 127.30.247.61:40489)
0925 05:06:19.913448 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:19.913454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:19.913474 (+ 20us) server_negotiation.cc:407] Connection header received
0925 05:06:19.913630 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:19.913636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:19.913715 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:19.913824 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:19.914623 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.915323 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.916471 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.916999 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.917136 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:19.920169 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:19.920175 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:19.920180 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:19.920220 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:19.923049 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:19.923709 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:19.923717 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:19.923721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:19.923847 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:19.924215 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:19.924222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:19.924225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:19.924500 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:19.924698 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:19.925079 (+ 381us) server_negotiation.cc:299] Negotiation successful
0925 05:06:19.925296 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 05:06:19.925623 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.913113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:40489 (local address 127.0.0.1:43818)
0925 05:06:19.913320 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.913345 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:19.913479 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:19.913869 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:19.913875 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:19.914281 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:19.914457 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:19.914466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.915460 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.915468 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:19.916314 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:19.916326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.917164 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.917170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:19.917395 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:19.918031 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:19.918049 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:19.919981 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:19.923202 (+ 3221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:19.923209 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:19.923213 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:19.923591 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:19.923962 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:19.923967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:19.923970 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:19.924106 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:19.924662 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:19.924667 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:19.924831 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 05:06:19.924975 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 05:06:19.925252 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 05:06:19.926724 21688 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:06:19.927757 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.927388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:43793 (local address 127.0.0.1:58414)
0925 05:06:19.927543 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.927633 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 05:06:19.928711 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.928341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:43793 (local address 127.0.0.1:58422)
0925 05:06:19.928478 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.928627 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 05:06:19.928892 21688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.60:43793: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
I20260925 05:06:19.970611 21762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } attempt: 3
I20260925 05:06:19.971978 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.971330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:43793 (local address 127.0.0.1:58434)
0925 05:06:19.971587 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.971729 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 05:06:19.973294 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.972830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:43793 (local address 127.0.0.1:58442)
0925 05:06:19.973071 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.973167 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260925 05:06:19.973511 21762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.60:43793: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
I20260925 05:06:19.980120 21688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } attempt: 1
I20260925 05:06:19.981127 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.980717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:43793 (local address 127.0.0.1:58456)
0925 05:06:19.980889 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.980976 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 05:06:19.982080 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.981770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:43793 (local address 127.0.0.1:58470)
0925 05:06:19.981912 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.981986 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 05:06:19.982281 21688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.60:43793: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
I20260925 05:06:20.050494 21688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } attempt: 2
I20260925 05:06:20.051740 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.051261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:43793 (local address 127.0.0.1:58486)
0925 05:06:20.051488 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:20.051586 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 05:06:20.052784 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.052436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:43793 (local address 127.0.0.1:58488)
0925 05:06:20.052603 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:20.052682 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 05:06:20.052979 21688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.60:43793: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
W20260925 05:06:20.092027 21769 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:06:20.092394 21769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:20.092556 21769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:20.092700 21769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:20.099746 21769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:20.099864 21769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:20.099964 21769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:20.100057 21769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:20.100145 21769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:20.109800 21769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:46773,127.30.247.61:40489,127.30.247.60:43793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.60.keytab
--principal=kudu/127.30.247.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.60:43793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.247.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:20.111074 21769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:20.112370 21769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:06:20.116731 21762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } attempt: 4
I20260925 05:06:20.118105 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.117418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:43793 (local address 127.0.0.1:58496)
0925 05:06:20.117743 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:20.117855 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 05:06:20.119189 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.118881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:43793 (local address 127.0.0.1:58508)
0925 05:06:20.119026 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:20.119095 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 05:06:20.119421 21762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.60:43793: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
W20260925 05:06:20.124528 21775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:20.124495 21774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:20.124605 21777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:20.125281 21769 server_base.cc:1034] running on GCE node
I20260925 05:06:20.126067 21769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:20.127417 21769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:20.128662 21769 hybrid_clock.cc:648] HybridClock initialized: now 1790312780128598 us; error 40 us; skew 500 ppm
I20260925 05:06:20.128976 21769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:20 dist-test-slave-5n86 krb5kdc[21608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312780, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:20.132014 21769 init.cc:377] Logged in from keytab as kudu/127.30.247.60@KRBTEST.COM (short username kudu)
I20260925 05:06:20.133478 21769 webserver.cc:460] Webserver started at http://127.30.247.60:39005/ using document root <none> and password file <none>
I20260925 05:06:20.134135 21769 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:20.134263 21769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:20.134581 21769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:20.137430 21769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-2/data/instance:
uuid: "9099e94444414720980fda8c563c8711"
format_stamp: "Formatted at 2026-09-25 05:06:20 on dist-test-slave-5n86"
server_key: "50c8c1fc6659d1c4ffcdccc03e970d4e"
server_key_iv: "c633c31b7727fef1c2a5a3394e5d122b"
server_key_version: "encryptionkey@0"
I20260925 05:06:20.138195 21769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-2/wal/instance:
uuid: "9099e94444414720980fda8c563c8711"
format_stamp: "Formatted at 2026-09-25 05:06:20 on dist-test-slave-5n86"
server_key: "50c8c1fc6659d1c4ffcdccc03e970d4e"
server_key_iv: "c633c31b7727fef1c2a5a3394e5d122b"
server_key_version: "encryptionkey@0"
I20260925 05:06:20.142982 21769 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 05:06:20.146284 21784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:20.147514 21769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:06:20.147691 21769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-2/wal
uuid: "9099e94444414720980fda8c563c8711"
format_stamp: "Formatted at 2026-09-25 05:06:20 on dist-test-slave-5n86"
server_key: "50c8c1fc6659d1c4ffcdccc03e970d4e"
server_key_iv: "c633c31b7727fef1c2a5a3394e5d122b"
server_key_version: "encryptionkey@0"
I20260925 05:06:20.147892 21769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:20.159179 21688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } attempt: 3
I20260925 05:06:20.160250 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.159795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:43793 (local address 127.0.0.1:58516)
0925 05:06:20.160012 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:20.160102 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 05:06:20.161315 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.160958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:43793 (local address 127.0.0.1:58520)
0925 05:06:20.161145 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:20.161212 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260925 05:06:20.161504 21688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.60:43793: Network error: Client connection negotiation failed: client connection to 127.30.247.60:43793: connect: Connection refused (error 111)
I20260925 05:06:20.165472 21769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:20.169714 21769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:20.169981 21769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:20.188086 21769 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.60:43793
I20260925 05:06:20.188105 21835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.60:43793 every 8 connection(s)
I20260925 05:06:20.189482 21769 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-2/data/info.pb
I20260925 05:06:20.192961 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:20.196966 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 21769
I20260925 05:06:20.197150 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312430481417-31708-0/minicluster-data/master-2/wal/instance
I20260925 05:06:20.197652 31708 external_mini_cluster.cc:1428] Setting key 7ae2ebd64c73fbeed5e7e6ea14bd2764
I20260925 05:06:20.200327 21836 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46773 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:06:20 dist-test-slave-5n86 krb5kdc[21608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312780, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.60@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
Sep 25 05:06:20 dist-test-slave-5n86 krb5kdc[21608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.60@KRBTEST.COM
I20260925 05:06:20.218698 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.204017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55478 (local address 127.30.247.62:46773)
0925 05:06:20.204248 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:20.204253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:20.204713 (+ 460us) server_negotiation.cc:407] Connection header received
0925 05:06:20.204911 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:20.204917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:20.204985 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:20.205086 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:20.206176 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.206834 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.208173 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.208780 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.208905 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:20.212125 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:20.212132 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:20.212137 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:20.212174 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:20.215465 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:20.216360 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:20.216367 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:20.216370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:20.216514 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:20.216975 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:20.216980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:20.216984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:20.217237 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:20.217399 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:20.218114 (+ 715us) server_negotiation.cc:299] Negotiation successful
0925 05:06:20.218327 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260925 05:06:20.219795 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.203914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:46773 (local address 127.0.0.1:55478)
0925 05:06:20.204581 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:20.204623 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:20.204779 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:20.205141 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:20.205148 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:20.205751 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:20.206013 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:20.206021 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.206987 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.206992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:20.208029 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:20.208040 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.209011 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.209019 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:20.209193 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:20.209806 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:20.209827 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:20.211911 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:20.215596 (+ 3685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:20.215608 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:20.215619 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:20.216220 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:20.216679 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:20.216688 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:20.216690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:20.216850 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:20.217384 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:20.217392 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:20.217697 (+ 305us) client_negotiation.cc:769] Sending connection context
0925 05:06:20.218005 (+ 308us) client_negotiation.cc:240] Negotiation successful
0925 05:06:20.218464 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":259,"threads_started":1}
I20260925 05:06:20.221099 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.199383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58528 (local address 127.30.247.60:43793)
0925 05:06:20.200123 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:20.200133 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:20.200174 (+ 41us) server_negotiation.cc:407] Connection header received
0925 05:06:20.202219 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:20.202240 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:20.202641 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:20.202930 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:20.204733 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.206020 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.207238 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.207878 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.208094 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:20.214185 (+ 6091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:20.214195 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:20.214209 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:20.214242 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:20.217698 (+ 3456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:20.218506 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:20.218513 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:20.218516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:20.218666 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:20.219077 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:20.219082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:20.219085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:20.219479 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:20.219797 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:20.220658 (+ 861us) server_negotiation.cc:299] Negotiation successful
0925 05:06:20.220830 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":354,"threads_started":1}
I20260925 05:06:20.221447 21836 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.61" port: 40489 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:06:20 dist-test-slave-5n86 krb5kdc[21608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312780, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.60@KRBTEST.COM for kudu/127.30.247.61@KRBTEST.COM
Sep 25 05:06:20 dist-test-slave-5n86 krb5kdc[21608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:06:20.236124 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.222208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:40489 (local address 127.0.0.1:43824)
0925 05:06:20.222377 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:20.222395 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:20.222491 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:20.222802 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:20.222806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:20.223238 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:20.223402 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:20.223413 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.224742 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.224749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:20.226025 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:20.226036 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.227008 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.227015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:20.227199 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:20.227735 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:20.227757 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:20.229944 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:20.233574 (+ 3630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:20.233580 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:20.233585 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:20.234026 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:20.234519 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:20.234522 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:20.234524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:20.234657 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:20.235180 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:20.235185 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:20.235327 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 05:06:20.235491 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 05:06:20.235755 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 05:06:20.236147 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.222315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43824 (local address 127.30.247.61:40489)
0925 05:06:20.222463 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:20.222467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:20.222482 (+ 15us) server_negotiation.cc:407] Connection header received
0925 05:06:20.222604 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:20.222607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:20.222676 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:20.222752 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:20.223563 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.224592 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.226184 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.226833 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.226954 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:20.230146 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:20.230154 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:20.230161 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:20.230202 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:20.233421 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:20.234149 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:20.234155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:20.234165 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:20.234373 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:20.234779 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:20.234785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:20.234788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:20.235057 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:20.235220 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:20.235583 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 05:06:20.235778 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 05:06:20.237404 21836 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:06:20.239777 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.223899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55480 (local address 127.30.247.62:46773)
0925 05:06:20.224074 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:20.224080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:20.224096 (+ 16us) server_negotiation.cc:407] Connection header received
0925 05:06:20.224147 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:20.224152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:20.224212 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:20.224300 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:20.225763 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.226764 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.228623 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.229029 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.229177 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:20.233919 (+ 4742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:20.233924 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:20.233927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:20.233968 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:20.237071 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:20.237751 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:20.237756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:20.237779 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:20.237956 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:20.238393 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:20.238400 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:20.238403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:20.238786 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:20.239059 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:20.239220 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 05:06:20.239419 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 05:06:20 dist-test-slave-5n86 krb5kdc[21608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312780, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.60@KRBTEST.COM for kudu/127.30.247.60@KRBTEST.COM
Sep 25 05:06:20 dist-test-slave-5n86 krb5kdc[21608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.61@KRBTEST.COM
I20260925 05:06:20.250478 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.238262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:43793 (local address 127.0.0.1:58542)
0925 05:06:20.238382 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:20.238398 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:20.238508 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:20.238792 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:20.238796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:20.239243 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:20.239403 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:20.239413 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.240491 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.240497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:20.241704 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:20.241715 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.242380 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.242385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:20.242604 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:20.243097 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:20.243113 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:20.244982 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:20.247901 (+ 2919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:20.247907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:20.247911 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:20.248351 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:20.248822 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:20.248830 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:20.248836 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:20.248988 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:20.249482 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:20.249486 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:20.249624 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 05:06:20.249773 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 05:06:20.250067 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":4}
I20260925 05:06:20.250511 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.238091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58542 (local address 127.30.247.60:43793)
0925 05:06:20.238240 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:20.238244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:20.238485 (+ 241us) server_negotiation.cc:407] Connection header received
0925 05:06:20.238611 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:20.238615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:20.238667 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:20.238736 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:20.239543 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.240354 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.241837 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.242239 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.242372 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:20.245158 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:20.245163 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:20.245166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:20.245193 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:20.247754 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:20.248484 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:20.248488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:20.248490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:20.248702 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:20.249101 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:20.249109 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:20.249113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:20.249365 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:20.249523 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:20.249856 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 05:06:20.250212 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":198,"server-negotiator.queue_time_us":52}
I20260925 05:06:20.255568 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.241427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43826 (local address 127.30.247.61:40489)
0925 05:06:20.241586 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:20.241591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:20.241607 (+ 16us) server_negotiation.cc:407] Connection header received
0925 05:06:20.241765 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:20.241769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:20.241835 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:20.241932 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:20.243231 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.243898 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.245140 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.245548 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.245674 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:20.249721 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:20.249727 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:20.249736 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:20.249764 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:20.253115 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:20.253692 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:20.253697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:20.253700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:20.253846 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:20.254204 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:20.254212 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:20.254218 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:20.254537 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:20.254800 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:20.255046 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 05:06:20.255267 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 05:06:20.256402 21836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9099e94444414720980fda8c563c8711: Bootstrap starting.
I20260925 05:06:20.259831 21836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9099e94444414720980fda8c563c8711: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:20.260586 31708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:06:20.261024 21836 log.cc:824] T 00000000000000000000000000000000 P 9099e94444414720980fda8c563c8711: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:20.263906 21836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9099e94444414720980fda8c563c8711: No bootstrap required, opened a new log
I20260925 05:06:20.263278 21663 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ce8b4a6e862f404693ff65c3c3c56353"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55480
I20260925 05:06:20.274688 21836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9099e94444414720980fda8c563c8711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8b4a6e862f404693ff65c3c3c56353" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46773 } } peers { permanent_uuid: "d85ddd55f67447ad8abc89c1c14c4412" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 40489 } } peers { permanent_uuid: "9099e94444414720980fda8c563c8711" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } }
I20260925 05:06:20.275094 21836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9099e94444414720980fda8c563c8711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9099e94444414720980fda8c563c8711, State: Initialized, Role: FOLLOWER
I20260925 05:06:20.275588 21836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9099e94444414720980fda8c563c8711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8b4a6e862f404693ff65c3c3c56353" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46773 } } peers { permanent_uuid: "d85ddd55f67447ad8abc89c1c14c4412" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 40489 } } peers { permanent_uuid: "9099e94444414720980fda8c563c8711" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } }
I20260925 05:06:20.276448 21842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9099e94444414720980fda8c563c8711 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8b4a6e862f404693ff65c3c3c56353" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46773 } } peers { permanent_uuid: "d85ddd55f67447ad8abc89c1c14c4412" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 40489 } } peers { permanent_uuid: "9099e94444414720980fda8c563c8711" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } } }
I20260925 05:06:20.276835 21842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9099e94444414720980fda8c563c8711 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:06:20.277725 21836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9099e94444414720980fda8c563c8711 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 05:06:20.287197 21853 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9099e94444414720980fda8c563c8711: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:06:20.287329 21853 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:06:20.284919 21769 master_runner.cc:186] Error getting master registration for 127.30.247.62:46773: OK, instance_id { permanent_uuid: "ce8b4a6e862f404693ff65c3c3c56353" instance_seqno: 1790312779204293 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 05:06:20.295961 21769 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 05:06:20.338716 21688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } attempt: 4
Sep 25 05:06:20 dist-test-slave-5n86 krb5kdc[21608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312779, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for kudu/127.30.247.60@KRBTEST.COM
I20260925 05:06:20.354449 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.339536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:43793 (local address 127.0.0.1:58556)
0925 05:06:20.339776 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:20.339797 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:20.339935 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:20.340400 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:20.340406 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:20.340942 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:20.341156 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:20.341169 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.342636 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.342641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:20.343581 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:20.343592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.344631 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.344637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:20.344853 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:20.345516 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:20.345533 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:20.347746 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:20.351717 (+ 3971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:20.351721 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:20.351723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:20.352128 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:20.352520 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:20.352524 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:20.352526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:20.352774 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:20.353468 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:20.353473 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:20.353639 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 05:06:20.353773 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 05:06:20.354042 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 05:06:20.354790 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.339835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58556 (local address 127.30.247.60:43793)
0925 05:06:20.340061 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:20.340068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:20.340090 (+ 22us) server_negotiation.cc:407] Connection header received
0925 05:06:20.340162 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:20.340167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:20.340246 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:20.340352 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:20.341315 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.342483 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.343730 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.344439 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.344599 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:20.347935 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:20.347943 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:20.347948 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:20.347992 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:20.351591 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:20.352244 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:20.352250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:20.352253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:20.352388 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:20.352884 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:20.352890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:20.352893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:20.353334 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:20.353496 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:20.353919 (+ 423us) server_negotiation.cc:299] Negotiation successful
0925 05:06:20.354145 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 05:06:20.359261 21688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353: Bootstrap starting.
I20260925 05:06:20.362537 21688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:20.363848 21688 log.cc:824] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:20.366493 21688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353: No bootstrap required, opened a new log
I20260925 05:06:20.371425 21688 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8b4a6e862f404693ff65c3c3c56353" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46773 } } peers { permanent_uuid: "d85ddd55f67447ad8abc89c1c14c4412" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 40489 } } peers { permanent_uuid: "9099e94444414720980fda8c563c8711" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } }
I20260925 05:06:20.371759 21688 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce8b4a6e862f404693ff65c3c3c56353, State: Initialized, Role: FOLLOWER
I20260925 05:06:20.372202 21688 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8b4a6e862f404693ff65c3c3c56353" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46773 } } peers { permanent_uuid: "d85ddd55f67447ad8abc89c1c14c4412" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 40489 } } peers { permanent_uuid: "9099e94444414720980fda8c563c8711" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } }
I20260925 05:06:20.373220 21856 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8b4a6e862f404693ff65c3c3c56353" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46773 } } peers { permanent_uuid: "d85ddd55f67447ad8abc89c1c14c4412" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 40489 } } peers { permanent_uuid: "9099e94444414720980fda8c563c8711" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } } }
I20260925 05:06:20.373560 21856 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:06:20.374343 21688 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:06:20.374483 21663 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 05:06:20.374714 21663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:20.378446 21663 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8b4a6e862f404693ff65c3c3c56353" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46773 } } peers { permanent_uuid: "d85ddd55f67447ad8abc89c1c14c4412" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 40489 } } peers { permanent_uuid: "9099e94444414720980fda8c563c8711" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } }
I20260925 05:06:20.381210 21811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce8b4a6e862f404693ff65c3c3c56353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9099e94444414720980fda8c563c8711"
I20260925 05:06:20.381692 21811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9099e94444414720980fda8c563c8711 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:20.385233 21811 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9099e94444414720980fda8c563c8711 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce8b4a6e862f404693ff65c3c3c56353 in term 1.
I20260925 05:06:20.385963 21638 leader_election.cc:302] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9099e94444414720980fda8c563c8711, ce8b4a6e862f404693ff65c3c3c56353; no voters:
I20260925 05:06:20.386969 21856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353 [term 1 FOLLOWER]: Leader election won for term 1
W20260925 05:06:20.391351 21867 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:06:20.391479 21867 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:06:20.386309 21663 leader_election.cc:288] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353 [CANDIDATE]: Term 1 election: Requested vote from peers d85ddd55f67447ad8abc89c1c14c4412 (127.30.247.61:40489), 9099e94444414720980fda8c563c8711 (127.30.247.60:43793)
I20260925 05:06:20.395960 21856 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353 [term 1 LEADER]: Becoming Leader. State: Replica: ce8b4a6e862f404693ff65c3c3c56353, State: Running, Role: LEADER
I20260925 05:06:20.396749 21856 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8b4a6e862f404693ff65c3c3c56353" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46773 } } peers { permanent_uuid: "d85ddd55f67447ad8abc89c1c14c4412" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 40489 } } peers { permanent_uuid: "9099e94444414720980fda8c563c8711" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } }
I20260925 05:06:20.402858 21868 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce8b4a6e862f404693ff65c3c3c56353. Latest consensus state: current_term: 1 leader_uuid: "ce8b4a6e862f404693ff65c3c3c56353" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8b4a6e862f404693ff65c3c3c56353" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46773 } } peers { permanent_uuid: "d85ddd55f67447ad8abc89c1c14c4412" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 40489 } } peers { permanent_uuid: "9099e94444414720980fda8c563c8711" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } } }
I20260925 05:06:20.403275 21868 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:20.406749 21621 master_runner.cc:186] Error getting master registration for 127.30.247.61:40489: OK, instance_id { permanent_uuid: "d85ddd55f67447ad8abc89c1c14c4412" instance_seqno: 1790312779691854 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 05:06:20.418640 21762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } attempt: 5
I20260925 05:06:20.419020 21621 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 05:06:20.419286 21873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:06:20.421622 21873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260925 05:06:20.422716 21790 tablet.cc:2369] T 00000000000000000000000000000000 P 9099e94444414720980fda8c563c8711: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 05:06:20 dist-test-slave-5n86 krb5kdc[21608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312779, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.61@KRBTEST.COM for kudu/127.30.247.60@KRBTEST.COM
I20260925 05:06:20.436678 21811 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9099e94444414720980fda8c563c8711 [term 1 FOLLOWER]: Refusing update from remote peer ce8b4a6e862f404693ff65c3c3c56353: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:20.437626 21868 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9099e94444414720980fda8c563c8711" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 05:06:20.444082 21642 tablet.cc:2369] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:06:20.456754 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.419253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:43793 (local address 127.0.0.1:58564)
0925 05:06:20.419454 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:20.419475 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:20.419599 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:20.420202 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:20.420208 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:20.420741 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:20.420950 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:20.420963 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.423534 (+ 2571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.423540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:20.424987 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:20.425002 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.426386 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.426392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:20.426606 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:20.427392 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:20.427438 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:20.447085 (+ 19647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:20.453791 (+ 6706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:20.453796 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:20.453800 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:20.454373 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:20.454840 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:20.454846 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:20.454849 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:20.455028 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:20.455673 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:20.455679 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:20.455837 (+ 158us) client_negotiation.cc:769] Sending connection context
0925 05:06:20.456001 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 05:06:20.456303 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 05:06:20.457847 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.419335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58564 (local address 127.30.247.60:43793)
0925 05:06:20.419489 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:20.419502 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:20.419528 (+ 26us) server_negotiation.cc:407] Connection header received
0925 05:06:20.419803 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:20.419811 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:20.419894 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:20.419984 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:20.421163 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.422347 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.425191 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.426022 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.426156 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:20.447560 (+ 21404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:20.447568 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:20.447573 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:20.447615 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:20.453634 (+ 6019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:20.454516 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:20.454521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:20.454525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:20.454699 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:20.455156 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:20.455161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:20.455165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:20.455478 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:20.455613 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:20.457246 (+ 1633us) server_negotiation.cc:299] Negotiation successful
0925 05:06:20.457471 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 05:06:20.465103 21762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d85ddd55f67447ad8abc89c1c14c4412: Bootstrap starting.
I20260925 05:06:20.469220 21762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d85ddd55f67447ad8abc89c1c14c4412: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:20.470587 21762 log.cc:824] T 00000000000000000000000000000000 P d85ddd55f67447ad8abc89c1c14c4412: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:20.474289 21842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9099e94444414720980fda8c563c8711 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce8b4a6e862f404693ff65c3c3c56353. Latest consensus state: current_term: 1 leader_uuid: "ce8b4a6e862f404693ff65c3c3c56353" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8b4a6e862f404693ff65c3c3c56353" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46773 } } peers { permanent_uuid: "d85ddd55f67447ad8abc89c1c14c4412" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 40489 } } peers { permanent_uuid: "9099e94444414720980fda8c563c8711" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } } }
I20260925 05:06:20.474623 21842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9099e94444414720980fda8c563c8711 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:06:20.477625 21876 mvcc.cc:205] Tried to move back new op lower bound from 7333121148655288320 to 7333121148517593088. Current Snapshot: MvccSnapshot[applied={T|T < 7333121148655288320}]
I20260925 05:06:20.479938 21762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d85ddd55f67447ad8abc89c1c14c4412: No bootstrap required, opened a new log
I20260925 05:06:20.492739 21856 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce8b4a6e862f404693ff65c3c3c56353" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8b4a6e862f404693ff65c3c3c56353" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46773 } } peers { permanent_uuid: "d85ddd55f67447ad8abc89c1c14c4412" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 40489 } } peers { permanent_uuid: "9099e94444414720980fda8c563c8711" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } } }
I20260925 05:06:20.493074 21856 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:20.496800 21879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.431361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58568 (local address 127.30.247.60:43793)
0925 05:06:20.438169 (+ 6808us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:20.438174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:20.438322 (+ 148us) server_negotiation.cc:407] Connection header received
0925 05:06:20.438698 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:20.438738 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:20.438855 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:20.439028 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:20.442628 (+ 3600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.444926 (+ 2298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.448030 (+ 3104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.451934 (+ 3904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.452044 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:20.476509 (+ 24465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:20.476517 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:20.476522 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:20.476596 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:20.486173 (+ 9577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:20.493232 (+ 7059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:20.493237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:20.493241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:20.493487 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:20.494036 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:20.494041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:20.494044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:20.494546 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:20.495893 (+ 1347us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:20.496032 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 05:06:20.496300 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6263,"thread_start_us":5878,"threads_started":1}
I20260925 05:06:20.498591 21873 catalog_manager.cc:1347] Generated new cluster ID: 086c4830a9c24f7b8c525d2a7db574dd
I20260925 05:06:20.498687 21873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:06:20.500631 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.457488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43834 (local address 127.30.247.61:40489)
0925 05:06:20.457674 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:20.457683 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:20.469117 (+ 11434us) server_negotiation.cc:407] Connection header received
0925 05:06:20.469367 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:20.469373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:20.469450 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:20.469517 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:20.471756 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.472809 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.477259 (+ 4450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.477957 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.478052 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:20.488833 (+ 10781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:20.488843 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:20.488850 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:20.488904 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:20.492937 (+ 4033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:20.493790 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:20.493798 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:20.493803 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:20.493956 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:20.494388 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:20.494393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:20.494396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:20.497898 (+ 3502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:20.498166 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:20.499968 (+ 1802us) server_negotiation.cc:299] Negotiation successful
0925 05:06:20.500211 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":64}
I20260925 05:06:20.503072 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.431591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55486 (local address 127.30.247.62:46773)
0925 05:06:20.431769 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:20.431775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:20.458268 (+ 26493us) server_negotiation.cc:407] Connection header received
0925 05:06:20.458712 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:20.458718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:20.458804 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:20.458879 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:20.466248 (+ 7369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.467418 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.471768 (+ 4350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.472412 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.472548 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:20.493896 (+ 21348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:20.493903 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:20.493908 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:20.493950 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:20.497532 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:20.498851 (+ 1319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:20.498856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:20.498860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:20.499062 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:20.499525 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:20.499530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:20.499533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:20.500309 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:20.500795 (+ 486us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:20.501793 (+ 998us) server_negotiation.cc:299] Negotiation successful
0925 05:06:20.502271 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 05:06:20.507113 21762 raft_consensus.cc:348] T 00000000000000000000000000000000 P d85ddd55f67447ad8abc89c1c14c4412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8b4a6e862f404693ff65c3c3c56353" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46773 } } peers { permanent_uuid: "d85ddd55f67447ad8abc89c1c14c4412" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 40489 } } peers { permanent_uuid: "9099e94444414720980fda8c563c8711" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } }
I20260925 05:06:20.507578 21762 raft_consensus.cc:729] T 00000000000000000000000000000000 P d85ddd55f67447ad8abc89c1c14c4412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d85ddd55f67447ad8abc89c1c14c4412, State: Initialized, Role: FOLLOWER
I20260925 05:06:20.508154 21762 consensus_queue.cc:260] T 00000000000000000000000000000000 P d85ddd55f67447ad8abc89c1c14c4412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8b4a6e862f404693ff65c3c3c56353" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46773 } } peers { permanent_uuid: "d85ddd55f67447ad8abc89c1c14c4412" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 40489 } } peers { permanent_uuid: "9099e94444414720980fda8c563c8711" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } }
I20260925 05:06:20.510403 21842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9099e94444414720980fda8c563c8711 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce8b4a6e862f404693ff65c3c3c56353" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8b4a6e862f404693ff65c3c3c56353" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46773 } } peers { permanent_uuid: "d85ddd55f67447ad8abc89c1c14c4412" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 40489 } } peers { permanent_uuid: "9099e94444414720980fda8c563c8711" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } } }
I20260925 05:06:20.510663 21842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9099e94444414720980fda8c563c8711 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:06:20.513878 21762 sys_catalog.cc:564] T 00000000000000000000000000000000 P d85ddd55f67447ad8abc89c1c14c4412 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:06:20.513909 21886 sys_catalog.cc:455] T 00000000000000000000000000000000 P d85ddd55f67447ad8abc89c1c14c4412 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8b4a6e862f404693ff65c3c3c56353" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46773 } } peers { permanent_uuid: "d85ddd55f67447ad8abc89c1c14c4412" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 40489 } } peers { permanent_uuid: "9099e94444414720980fda8c563c8711" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } } }
I20260925 05:06:20.514339 21886 sys_catalog.cc:458] T 00000000000000000000000000000000 P d85ddd55f67447ad8abc89c1c14c4412 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:06:20.514503 21736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d85ddd55f67447ad8abc89c1c14c4412 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:20.518224 21736 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d85ddd55f67447ad8abc89c1c14c4412 [term 1 FOLLOWER]: Refusing update from remote peer ce8b4a6e862f404693ff65c3c3c56353: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:20.519091 21868 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d85ddd55f67447ad8abc89c1c14c4412" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 40489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:20.513945 21737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce8b4a6e862f404693ff65c3c3c56353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d85ddd55f67447ad8abc89c1c14c4412"
I20260925 05:06:20.524623 21737 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d85ddd55f67447ad8abc89c1c14c4412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce8b4a6e862f404693ff65c3c3c56353 in term 1.
I20260925 05:06:20.532020 21873 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:06:20.533882 21873 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:06:20.569603 21873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353: Generated new TSK 0
I20260925 05:06:20.570871 21873 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:06:20.585441 21886 sys_catalog.cc:455] T 00000000000000000000000000000000 P d85ddd55f67447ad8abc89c1c14c4412 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce8b4a6e862f404693ff65c3c3c56353. Latest consensus state: current_term: 1 leader_uuid: "ce8b4a6e862f404693ff65c3c3c56353" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8b4a6e862f404693ff65c3c3c56353" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46773 } } peers { permanent_uuid: "d85ddd55f67447ad8abc89c1c14c4412" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 40489 } } peers { permanent_uuid: "9099e94444414720980fda8c563c8711" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } } }
I20260925 05:06:20.585765 21886 sys_catalog.cc:458] T 00000000000000000000000000000000 P d85ddd55f67447ad8abc89c1c14c4412 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:06:20.586095 21896 sys_catalog.cc:455] T 00000000000000000000000000000000 P d85ddd55f67447ad8abc89c1c14c4412 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce8b4a6e862f404693ff65c3c3c56353" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8b4a6e862f404693ff65c3c3c56353" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46773 } } peers { permanent_uuid: "d85ddd55f67447ad8abc89c1c14c4412" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 40489 } } peers { permanent_uuid: "9099e94444414720980fda8c563c8711" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } } }
I20260925 05:06:20.586426 21896 sys_catalog.cc:458] T 00000000000000000000000000000000 P d85ddd55f67447ad8abc89c1c14c4412 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:06:20.587616 21856 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce8b4a6e862f404693ff65c3c3c56353" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8b4a6e862f404693ff65c3c3c56353" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 46773 } } peers { permanent_uuid: "d85ddd55f67447ad8abc89c1c14c4412" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 40489 } } peers { permanent_uuid: "9099e94444414720980fda8c563c8711" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 43793 } } }
I20260925 05:06:20.587913 21856 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce8b4a6e862f404693ff65c3c3c56353 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:20.617004 21903 catalog_manager.cc:1259] Loaded cluster ID: 086c4830a9c24f7b8c525d2a7db574dd
I20260925 05:06:20.617120 21903 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d85ddd55f67447ad8abc89c1c14c4412: loading cluster ID for follower catalog manager: success
I20260925 05:06:20.620906 21903 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d85ddd55f67447ad8abc89c1c14c4412: acquiring CA information for follower catalog manager: success
I20260925 05:06:20.623327 21903 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d85ddd55f67447ad8abc89c1c14c4412: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 05:06:20.730719 21692 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:20.693003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55498 (local address 127.30.247.62:46773)
0925 05:06:20.693222 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:20.693229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:20.728583 (+ 35354us) server_negotiation.cc:407] Connection header received
0925 05:06:20.729105 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:20.729114 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:20.729210 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:20.729284 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:20.730441 (+ 1157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55498: BlockingRecv error: recv got EOF from 127.0.0.1:55498 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 05:06:20.732872 21879 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:20.714319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58580 (local address 127.30.247.60:43793)
0925 05:06:20.714550 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:20.714559 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:20.731290 (+ 16731us) server_negotiation.cc:407] Connection header received
0925 05:06:20.731536 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:20.731543 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:20.731640 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:20.731717 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:20.732631 (+ 914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58580: BlockingRecv error: recv got EOF from 127.0.0.1:58580 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 05:06:20.743124 21766 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:20.741058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43848 (local address 127.30.247.61:40489)
0925 05:06:20.741279 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:20.741286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:20.741658 (+ 372us) server_negotiation.cc:407] Connection header received
0925 05:06:20.741870 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:20.741875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:20.741957 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:20.742035 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:20.742906 (+ 871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43848: BlockingRecv error: recv got EOF from 127.0.0.1:43848 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 05:06:20.784439 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.769006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43850 (local address 127.30.247.61:40489)
0925 05:06:20.769177 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:20.769182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:20.777129 (+ 7947us) server_negotiation.cc:407] Connection header received
0925 05:06:20.777359 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:20.777364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:20.777542 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:20.777686 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:20.778441 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.779629 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.781658 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.782300 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.782382 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:20.783054 (+ 672us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:20.783665 (+ 611us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:20.783777 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:20.784005 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 05:06:20.784095 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 05:06:20.834297 21879 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:20.821306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58594 (local address 127.30.247.60:43793)
0925 05:06:20.821505 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:20.821512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:20.832162 (+ 10650us) server_negotiation.cc:407] Connection header received
0925 05:06:20.832745 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:20.832751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:20.832887 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:20.832962 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:20.834011 (+ 1049us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58594: BlockingRecv error: recv got EOF from 127.0.0.1:58594 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 05:06:20.898989 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.881199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43854 (local address 127.30.247.61:40489)
0925 05:06:20.881418 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:20.881424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:20.891988 (+ 10564us) server_negotiation.cc:407] Connection header received
0925 05:06:20.892507 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:20.892514 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:20.892625 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:20.892697 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:20.893398 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.894598 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.896694 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.897390 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.897495 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:20.898107 (+ 612us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:20.898292 (+ 185us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:20.898357 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:20.898595 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 05:06:20.898686 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 05:06:20.910619 21879 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:20.908313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58596 (local address 127.30.247.60:43793)
0925 05:06:20.908500 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:20.908506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:20.908990 (+ 484us) server_negotiation.cc:407] Connection header received
0925 05:06:20.909254 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:20.909260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:20.909354 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:20.909425 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:20.910386 (+ 961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58596: BlockingRecv error: recv got EOF from 127.0.0.1:58596 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 05:06:20.923364 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.916566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43866 (local address 127.30.247.61:40489)
0925 05:06:20.916729 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:20.916735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:20.917216 (+ 481us) server_negotiation.cc:407] Connection header received
0925 05:06:20.917396 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:20.917404 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:20.917495 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:20.917618 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:20.918079 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.919302 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.921236 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.921928 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.922059 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:20.922504 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:20.922665 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:20.922762 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:20.922919 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 05:06:20.923002 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 05:06:20.934512 21879 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:20.930687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58602 (local address 127.30.247.60:43793)
0925 05:06:20.930875 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:20.930881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:20.932983 (+ 2102us) server_negotiation.cc:407] Connection header received
0925 05:06:20.933154 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:20.933159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:20.933276 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:20.933391 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:20.934264 (+ 873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58602: BlockingRecv error: recv got EOF from 127.0.0.1:58602 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:06:20.952610 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.945872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43868 (local address 127.30.247.61:40489)
0925 05:06:20.946081 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:20.946087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:20.946587 (+ 500us) server_negotiation.cc:407] Connection header received
0925 05:06:20.946747 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:20.946753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:20.946872 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:20.946966 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:20.947379 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.948470 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.950444 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.951137 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.951268 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:20.951783 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:20.951911 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:20.951997 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:20.952196 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 05:06:20.952280 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
W20260925 05:06:20.961781 21879 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:20.959933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58604 (local address 127.30.247.60:43793)
0925 05:06:20.960146 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:20.960152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:20.960172 (+ 20us) server_negotiation.cc:407] Connection header received
0925 05:06:20.960334 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:20.960340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:20.960428 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:20.960526 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:20.961552 (+ 1026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58604: BlockingRecv error: recv got EOF from 127.0.0.1:58604 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
I20260925 05:06:20.980711 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.973851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43882 (local address 127.30.247.61:40489)
0925 05:06:20.974037 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:20.974043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:20.974551 (+ 508us) server_negotiation.cc:407] Connection header received
0925 05:06:20.974732 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:20.974737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:20.974822 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:20.974916 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:20.975398 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.976557 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.978516 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.979214 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.979327 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:20.979749 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:20.979873 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:20.979959 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:20.980157 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 05:06:20.980254 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 05:06:20.992362 21879 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:20.990302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58616 (local address 127.30.247.60:43793)
0925 05:06:20.990470 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:20.990475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:20.990996 (+ 521us) server_negotiation.cc:407] Connection header received
0925 05:06:20.991175 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:20.991179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:20.991249 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:20.991334 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:20.992133 (+ 799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58616: BlockingRecv error: recv got EOF from 127.0.0.1:58616 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 05:06:21.018337 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:21.012148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43898 (local address 127.30.247.61:40489)
0925 05:06:21.012315 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:21.012320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:21.012716 (+ 396us) server_negotiation.cc:407] Connection header received
0925 05:06:21.012889 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:21.012893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:21.012962 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:21.013059 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:21.013533 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.014474 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.016581 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.017013 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.017119 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:21.017672 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:21.017775 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:21.017841 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:21.018002 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 05:06:21.018072 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 05:06:21.026279 21879 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:21.024124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58628 (local address 127.30.247.60:43793)
0925 05:06:21.024330 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:21.024336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:21.024723 (+ 387us) server_negotiation.cc:407] Connection header received
0925 05:06:21.024877 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:21.024881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:21.024958 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:21.025051 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:21.026028 (+ 977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58628: BlockingRecv error: recv got EOF from 127.0.0.1:58628 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 05:06:21.068655 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:21.062299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43902 (local address 127.30.247.61:40489)
0925 05:06:21.062561 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:21.062569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:21.063165 (+ 596us) server_negotiation.cc:407] Connection header received
0925 05:06:21.063384 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:21.063389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:21.063458 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:21.063568 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:21.063997 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.064869 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.066742 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.067222 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.067324 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:21.067839 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:21.067934 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:21.068005 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:21.068186 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 05:06:21.068249 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260925 05:06:21.077477 21879 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:21.075270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58630 (local address 127.30.247.60:43793)
0925 05:06:21.075556 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:21.075565 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:21.076075 (+ 510us) server_negotiation.cc:407] Connection header received
0925 05:06:21.076195 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:21.076200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:21.076295 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:21.076402 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:21.077193 (+ 791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58630: BlockingRecv error: recv got EOF from 127.0.0.1:58630 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 05:06:21.151441 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:21.145276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43912 (local address 127.30.247.61:40489)
0925 05:06:21.145495 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:21.145501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:21.145855 (+ 354us) server_negotiation.cc:407] Connection header received
0925 05:06:21.146050 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:21.146054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:21.146122 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:21.146208 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:21.146696 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.147442 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.149475 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.149947 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.150058 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:21.150596 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:21.150719 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:21.150794 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:21.150991 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 05:06:21.151064 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260925 05:06:21.159226 21879 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:21.157307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58646 (local address 127.30.247.60:43793)
0925 05:06:21.157492 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:21.157498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:21.157859 (+ 361us) server_negotiation.cc:407] Connection header received
0925 05:06:21.158091 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:21.158096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:21.158181 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:21.158280 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:21.159024 (+ 744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58646: BlockingRecv error: recv got EOF from 127.0.0.1:58646 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 05:06:21.289564 21853 catalog_manager.cc:1259] Loaded cluster ID: 086c4830a9c24f7b8c525d2a7db574dd
I20260925 05:06:21.289682 21853 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9099e94444414720980fda8c563c8711: loading cluster ID for follower catalog manager: success
I20260925 05:06:21.292840 21853 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9099e94444414720980fda8c563c8711: acquiring CA information for follower catalog manager: success
I20260925 05:06:21.295078 21853 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9099e94444414720980fda8c563c8711: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:06:21.296379 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:21.290653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43926 (local address 127.30.247.61:40489)
0925 05:06:21.290862 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:21.290868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:21.291208 (+ 340us) server_negotiation.cc:407] Connection header received
0925 05:06:21.291395 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:21.291400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:21.291470 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:21.291557 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:21.292047 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.292815 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.294752 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.295191 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.295312 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:21.295690 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:21.295804 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:21.295873 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:21.296011 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 05:06:21.296072 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 05:06:21.307363 21879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:21.301901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58656 (local address 127.30.247.60:43793)
0925 05:06:21.302140 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:21.302147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:21.302458 (+ 311us) server_negotiation.cc:407] Connection header received
0925 05:06:21.302680 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:21.302684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:21.302802 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:21.302919 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:21.303312 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.304025 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.305338 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.305735 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.305865 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:21.306341 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:21.306791 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:21.306952 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:21.307030 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 05:06:21.307107 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 05:06:21.325961 21879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:21.318291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58668 (local address 127.30.247.60:43793)
0925 05:06:21.318455 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:21.318461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:21.319858 (+ 1397us) server_negotiation.cc:407] Connection header received
0925 05:06:21.320118 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:21.320123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:21.320215 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:21.320294 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:21.320957 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.321879 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.323934 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.324611 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.324753 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:21.325222 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:21.325342 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:21.325441 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:21.325581 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 05:06:21.325664 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 05:06:21.326021 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:21.319378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43940 (local address 127.30.247.61:40489)
0925 05:06:21.319537 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:21.319541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:21.320182 (+ 641us) server_negotiation.cc:407] Connection header received
0925 05:06:21.320434 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:21.320442 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:21.320526 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:21.320622 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:21.321388 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.322281 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.324003 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.324693 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.324802 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:21.325291 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:21.325400 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:21.325485 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:21.325697 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 05:06:21.325771 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:06:21.326290 22007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:21.318178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55506 (local address 127.30.247.62:46773)
0925 05:06:21.318864 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:21.318872 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:21.319322 (+ 450us) server_negotiation.cc:407] Connection header received
0925 05:06:21.319441 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:21.319448 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:21.319644 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:21.319825 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:21.320523 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.321577 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.323771 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.324292 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.324431 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:21.324968 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:21.325402 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:21.325501 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:21.325908 (+ 407us) server_negotiation.cc:299] Negotiation successful
0925 05:06:21.325986 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":377,"threads_started":1}
I20260925 05:06:21.330935 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 21621
I20260925 05:06:21.331902 21683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:06:21.439966 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 21621
I20260925 05:06:21.460273 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 21695
I20260925 05:06:21.461534 21757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:06:21.567165 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 21695
I20260925 05:06:21.587172 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 21769
I20260925 05:06:21.588294 21831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:06:21.694154 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 21769
2026-09-25T05:06:21Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4246 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:06:21 dist-test-slave-5n86 krb5kdc[22015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:06:21 dist-test-slave-5n86 krb5kdc[22015](info): set up 2 sockets
Sep 25 05:06:21 dist-test-slave-5n86 krb5kdc[22015](info): commencing operation
krb5kdc: starting...
W20260925 05:06:22.801347 31708 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/krb5kdc/test-user.keytab.
Sep 25 05:06:22 dist-test-slave-5n86 krb5kdc[22015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:06:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:06:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
Entry for principal kudu/127.30.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab.
I20260925 05:06:23.040086 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:45713
--webserver_interface=127.30.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:38441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:45713,127.30.247.61:33401,127.30.247.60:37501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:23.383334 22028 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:06:23.383718 22028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:23.383842 22028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:23.383980 22028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:23.391119 22028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:23.391254 22028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:23.391326 22028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:23.391377 22028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:23.391431 22028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:23.400933 22028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:38441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:45713,127.30.247.61:33401,127.30.247.60:37501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.62.keytab
--principal=kudu/127.30.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.62:45713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:23.402201 22028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:23.403427 22028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:23.414875 22036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:23.414932 22033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:23.414961 22034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:23.415972 22028 server_base.cc:1034] running on GCE node
I20260925 05:06:23.416780 22028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:23.418090 22028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:23.419348 22028 hybrid_clock.cc:648] HybridClock initialized: now 1790312783419286 us; error 33 us; skew 500 ppm
I20260925 05:06:23.419674 22028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:23 dist-test-slave-5n86 krb5kdc[22015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312783, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:23.422791 22028 init.cc:377] Logged in from keytab as kudu/127.30.247.62@KRBTEST.COM (short username kudu)
I20260925 05:06:23.424216 22028 webserver.cc:460] Webserver started at http://127.30.247.62:41579/ using document root <none> and password file <none>
I20260925 05:06:23.424950 22028 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:23.425082 22028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:23.425390 22028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:23.428222 22028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-0/data/instance:
uuid: "e5857057b34d4b1da6685e0aa5ef97b0"
format_stamp: "Formatted at 2026-09-25 05:06:23 on dist-test-slave-5n86"
server_key: "caca32318640a888d4c8c6b0b658dd92"
server_key_iv: "98a810e1faf0d20c7f91625ebae35de0"
server_key_version: "encryptionkey@0"
I20260925 05:06:23.429046 22028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-0/wal/instance:
uuid: "e5857057b34d4b1da6685e0aa5ef97b0"
format_stamp: "Formatted at 2026-09-25 05:06:23 on dist-test-slave-5n86"
server_key: "caca32318640a888d4c8c6b0b658dd92"
server_key_iv: "98a810e1faf0d20c7f91625ebae35de0"
server_key_version: "encryptionkey@0"
I20260925 05:06:23.433974 22028 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 05:06:23.437353 22043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:23.438619 22028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:06:23.438804 22028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-0/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-0/wal
uuid: "e5857057b34d4b1da6685e0aa5ef97b0"
format_stamp: "Formatted at 2026-09-25 05:06:23 on dist-test-slave-5n86"
server_key: "caca32318640a888d4c8c6b0b658dd92"
server_key_iv: "98a810e1faf0d20c7f91625ebae35de0"
server_key_version: "encryptionkey@0"
I20260925 05:06:23.439010 22028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:23.470551 22028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:23.474987 22028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:23.475275 22028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:23.493630 22028 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.62:45713
I20260925 05:06:23.493649 22094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.62:45713 every 8 connection(s)
I20260925 05:06:23.495370 22028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-0/data/info.pb
I20260925 05:06:23.499018 22095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:23.503113 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 22028
I20260925 05:06:23.503298 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-0/wal/instance
I20260925 05:06:23.503837 31708 external_mini_cluster.cc:1428] Setting key e0e0181bac6a82a2fee2ec9a9c72f7b8
I20260925 05:06:23.506788 22095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45713 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:06:23 dist-test-slave-5n86 krb5kdc[22015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312783, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.247.61@KRBTEST.COM; defaulting to no policy
I20260925 05:06:23.532282 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:23.510504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:45713 (local address 127.0.0.1:38126)
0925 05:06:23.512149 (+ 1645us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:23.512190 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:23.514348 (+ 2158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:23.515215 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:23.515224 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:23.516010 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:23.516502 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:23.516527 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:23.518084 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:23.518093 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:23.519953 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:23.519969 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:23.521035 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:23.521044 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:23.521272 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:23.521969 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:23.522003 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:23.524755 (+ 2752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:23.527962 (+ 3207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:23.527976 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:23.527987 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:23.528382 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:23.528849 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:23.528853 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:23.528857 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:23.529006 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:23.529771 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:23.529780 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:23.529989 (+ 209us) client_negotiation.cc:769] Sending connection context
0925 05:06:23.530288 (+ 299us) client_negotiation.cc:240] Negotiation successful
0925 05:06:23.530953 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1409,"mutex_wait_us":185,"thread_start_us":1226,"threads_started":1}
I20260925 05:06:23.532337 22098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:23.510505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38126 (local address 127.30.247.62:45713)
0925 05:06:23.511659 (+ 1154us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:23.511677 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:23.512345 (+ 668us) server_negotiation.cc:407] Connection header received
0925 05:06:23.514740 (+ 2395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:23.514746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:23.514969 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:23.515127 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:23.516750 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:23.517911 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:23.520174 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:23.520801 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:23.520933 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:23.524973 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:23.524980 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:23.524991 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:23.525019 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:23.527828 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:23.528528 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:23.528572 (+ 44us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:23.528576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:23.528724 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:23.529122 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:23.529126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:23.529134 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:23.529612 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:23.529825 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:23.530422 (+ 597us) server_negotiation.cc:299] Negotiation successful
0925 05:06:23.530833 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":876,"thread_start_us":570,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.61@KRBTEST.COM" created.
I20260925 05:06:23.535756 22095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.61" port: 33401 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:06:23.536885 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:23.536397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:33401 (local address 127.0.0.1:52836)
0925 05:06:23.536607 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:23.536738 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 05:06:23.537196 22045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.247.61:33401: connect: Connection refused (error 111)
I20260925 05:06:23.538789 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:23.538384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:33401 (local address 127.0.0.1:52850)
0925 05:06:23.538546 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:23.538627 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 05:06:23.539062 22095 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.61:33401: Network error: Client connection negotiation failed: client connection to 127.30.247.61:33401: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.61.keytab.
Entry for principal kudu/127.30.247.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.61.keytab.
I20260925 05:06:23.565021 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.61:33401
--webserver_interface=127.30.247.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.61.keytab
--principal=kudu/127.30.247.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:38441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:45713,127.30.247.61:33401,127.30.247.60:37501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:06:23.588399 22095 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.61" port: 33401 } attempt: 1
I20260925 05:06:23.589679 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:23.589204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:33401 (local address 127.0.0.1:52864)
0925 05:06:23.589416 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:23.589544 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 05:06:23.590667 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:23.590303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:33401 (local address 127.0.0.1:52878)
0925 05:06:23.590445 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:23.590518 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 05:06:23.590876 22095 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.61:33401: Network error: Client connection negotiation failed: client connection to 127.30.247.61:33401: connect: Connection refused (error 111)
I20260925 05:06:23.659165 22095 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.61" port: 33401 } attempt: 2
I20260925 05:06:23.660363 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:23.659893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:33401 (local address 127.0.0.1:52890)
0925 05:06:23.660106 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:23.660207 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 05:06:23.661685 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:23.661259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:33401 (local address 127.0.0.1:52892)
0925 05:06:23.661435 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:23.661536 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260925 05:06:23.662001 22095 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.61:33401: Network error: Client connection negotiation failed: client connection to 127.30.247.61:33401: connect: Connection refused (error 111)
I20260925 05:06:23.753283 22095 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.61" port: 33401 } attempt: 3
I20260925 05:06:23.754401 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:23.753958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:33401 (local address 127.0.0.1:52906)
0925 05:06:23.754170 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:23.754269 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 05:06:23.755432 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:23.755074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:33401 (local address 127.0.0.1:52908)
0925 05:06:23.755217 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:23.755304 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 05:06:23.755681 22095 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.61:33401: Network error: Client connection negotiation failed: client connection to 127.30.247.61:33401: connect: Connection refused (error 111)
W20260925 05:06:23.886636 22102 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:06:23.886937 22102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:23.887076 22102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:23.887212 22102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:23.894331 22102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:23.894414 22102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:23.894601 22102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:23.894690 22102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:23.894781 22102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:23.898941 22095 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.61" port: 33401 } attempt: 4
I20260925 05:06:23.900218 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:23.899671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:33401 (local address 127.0.0.1:52914)
0925 05:06:23.899999 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:23.900088 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260925 05:06:23.901269 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:23.900922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:33401 (local address 127.0.0.1:52924)
0925 05:06:23.901067 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:23.901151 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.61:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 05:06:23.901474 22095 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.61:33401: Network error: Client connection negotiation failed: client connection to 127.30.247.61:33401: connect: Connection refused (error 111)
I20260925 05:06:23.904989 22102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:38441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:45713,127.30.247.61:33401,127.30.247.60:37501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.61.keytab
--principal=kudu/127.30.247.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.61:33401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.247.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:23.906243 22102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:23.907586 22102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:23.918781 22108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:23.918788 22107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:23.919246 22110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:23.919654 22102 server_base.cc:1034] running on GCE node
I20260925 05:06:23.920359 22102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:23.921694 22102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:23.922952 22102 hybrid_clock.cc:648] HybridClock initialized: now 1790312783922899 us; error 50 us; skew 500 ppm
I20260925 05:06:23.923297 22102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:23 dist-test-slave-5n86 krb5kdc[22015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312783, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:23.926339 22102 init.cc:377] Logged in from keytab as kudu/127.30.247.61@KRBTEST.COM (short username kudu)
I20260925 05:06:23.927745 22102 webserver.cc:460] Webserver started at http://127.30.247.61:37403/ using document root <none> and password file <none>
I20260925 05:06:23.928416 22102 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:23.928545 22102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:23.928875 22102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:23.931696 22102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-1/data/instance:
uuid: "35ed88e7a9524194b28f374016e47405"
format_stamp: "Formatted at 2026-09-25 05:06:23 on dist-test-slave-5n86"
server_key: "560cd43d4b78a9d84b1b0672373425c9"
server_key_iv: "bb9b3494247dc778b656c311e008f519"
server_key_version: "encryptionkey@0"
I20260925 05:06:23.932504 22102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-1/wal/instance:
uuid: "35ed88e7a9524194b28f374016e47405"
format_stamp: "Formatted at 2026-09-25 05:06:23 on dist-test-slave-5n86"
server_key: "560cd43d4b78a9d84b1b0672373425c9"
server_key_iv: "bb9b3494247dc778b656c311e008f519"
server_key_version: "encryptionkey@0"
I20260925 05:06:23.937547 22102 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260925 05:06:23.941010 22117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:23.942287 22102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 05:06:23.942463 22102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-1/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-1/wal
uuid: "35ed88e7a9524194b28f374016e47405"
format_stamp: "Formatted at 2026-09-25 05:06:23 on dist-test-slave-5n86"
server_key: "560cd43d4b78a9d84b1b0672373425c9"
server_key_iv: "bb9b3494247dc778b656c311e008f519"
server_key_version: "encryptionkey@0"
I20260925 05:06:23.942724 22102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:23.961856 22102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:23.966872 22102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:23.967178 22102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:23.985673 22102 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.61:33401
I20260925 05:06:23.985710 22168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.61:33401 every 8 connection(s)
I20260925 05:06:23.986991 22102 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-1/data/info.pb
I20260925 05:06:23.988291 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 22102
I20260925 05:06:23.988477 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-1/wal/instance
I20260925 05:06:23.989122 31708 external_mini_cluster.cc:1428] Setting key 7c26fe17615283f261312c581d1e0fe3
I20260925 05:06:23.991472 22169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:24.000941 22169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45713 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:06:24 dist-test-slave-5n86 krb5kdc[22015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312783, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.61@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.247.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.247.60@KRBTEST.COM" created.
I20260925 05:06:24.021674 22098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.003573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38134 (local address 127.30.247.62:45713)
0925 05:06:24.003758 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:24.003762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:24.005069 (+ 1307us) server_negotiation.cc:407] Connection header received
0925 05:06:24.006932 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:24.006940 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:24.007032 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:24.007154 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:24.008622 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.009313 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.010806 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.011287 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.011420 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.015754 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:24.015759 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:24.015762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:24.015793 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:24.018617 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.019360 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.019365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.019374 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.019495 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.019921 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.019926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.019934 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.020221 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:24.020391 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:24.021139 (+ 748us) server_negotiation.cc:299] Negotiation successful
0925 05:06:24.021359 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 05:06:24.022619 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.004084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:45713 (local address 127.0.0.1:38134)
0925 05:06:24.004892 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.004936 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:24.006657 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:24.007379 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:24.007387 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:24.007894 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:24.008420 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:24.008436 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.009476 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.009487 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:24.010652 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:24.010664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.011564 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.011572 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:24.011898 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.012802 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:24.012839 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:24.015525 (+ 2686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:24.018765 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:24.018771 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:24.018783 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:24.019218 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:24.019613 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:24.019619 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:24.019622 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:24.019796 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:24.020352 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:24.020360 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:24.020769 (+ 409us) client_negotiation.cc:769] Sending connection context
0925 05:06:24.021031 (+ 262us) client_negotiation.cc:240] Negotiation successful
0925 05:06:24.021523 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":322,"threads_started":1}
I20260925 05:06:24.024461 22169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.61" port: 33401 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:06:24 dist-test-slave-5n86 krb5kdc[22015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312783, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.61@KRBTEST.COM for kudu/127.30.247.61@KRBTEST.COM
I20260925 05:06:24.041080 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.025176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:33401 (local address 127.0.0.1:52940)
0925 05:06:24.025369 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.025388 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:24.025498 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:24.026621 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:24.026624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:24.026983 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:24.027121 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:24.027129 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.028627 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.028636 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:24.030233 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:24.030250 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.031083 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.031091 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:24.031313 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.031869 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:24.031896 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:24.034430 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:24.038717 (+ 4287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:24.038720 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:24.038723 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:24.039058 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:24.039472 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:24.039475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:24.039477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:24.039618 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:24.040359 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:24.040362 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:24.040466 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 05:06:24.040649 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 05:06:24.040828 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 05:06:24.041632 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.025424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52940 (local address 127.30.247.61:33401)
0925 05:06:24.026127 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:24.026133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:24.026157 (+ 24us) server_negotiation.cc:407] Connection header received
0925 05:06:24.026199 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:24.026204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:24.026426 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:24.026615 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:24.027304 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.028435 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.030393 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.030921 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.031022 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.034645 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:24.034661 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:24.034680 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:24.034731 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:24.038581 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.039195 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.039206 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.039212 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.039363 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.039735 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.039743 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.039750 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.040220 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:24.040437 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:24.040812 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 05:06:24.041105 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":425,"threads_started":1}
I20260925 05:06:24.044086 22169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:06:24.045215 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.044801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:37501 (local address 127.0.0.1:41204)
0925 05:06:24.044973 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.045083 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 05:06:24.045483 22118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
I20260925 05:06:24.047205 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.046767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:37501 (local address 127.0.0.1:41218)
0925 05:06:24.046944 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.047025 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260925 05:06:24.047451 22169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.60:37501: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.247.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.60.keytab.
Entry for principal kudu/127.30.247.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.60.keytab.
I20260925 05:06:24.055969 31708 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43xS0j/build/asan/bin/kudu
/tmp/dist-test-task43xS0j/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.60:37501
--webserver_interface=127.30.247.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.60.keytab
--principal=kudu/127.30.247.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.247.20:38441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.247.62:45713,127.30.247.61:33401,127.30.247.60:37501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:06:24.096776 22169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } attempt: 1
I20260925 05:06:24.098047 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.097470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:37501 (local address 127.0.0.1:41232)
0925 05:06:24.097716 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.097856 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 05:06:24.099174 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.098740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:37501 (local address 127.0.0.1:41244)
0925 05:06:24.098925 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.099034 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 05:06:24.099392 22169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.60:37501: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
I20260925 05:06:24.167654 22169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } attempt: 2
I20260925 05:06:24.168891 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.168337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:37501 (local address 127.0.0.1:41258)
0925 05:06:24.168595 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.168705 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 05:06:24.169888 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.169528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:37501 (local address 127.0.0.1:41268)
0925 05:06:24.169673 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.169755 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 05:06:24.170113 22169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.60:37501: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
I20260925 05:06:24.200749 22095 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.61" port: 33401 } attempt: 5
Sep 25 05:06:24 dist-test-slave-5n86 krb5kdc[22015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312783, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for kudu/127.30.247.61@KRBTEST.COM
I20260925 05:06:24.215993 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.201854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52944 (local address 127.30.247.61:33401)
0925 05:06:24.202034 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:24.202039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:24.202060 (+ 21us) server_negotiation.cc:407] Connection header received
0925 05:06:24.202126 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:24.202129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:24.202206 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:24.202294 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:24.203416 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.204148 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.205856 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.206315 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.206420 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.210421 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:24.210428 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:24.210434 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:24.210472 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:24.213169 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.214088 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.214092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.214094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.214217 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.214692 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.214695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.214697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.214956 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:24.215076 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:24.215616 (+ 540us) server_negotiation.cc:299] Negotiation successful
0925 05:06:24.215784 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 05:06:24.216405 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.201515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:33401 (local address 127.0.0.1:52944)
0925 05:06:24.201808 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.201835 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:24.201997 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:24.202393 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:24.202400 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:24.202958 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:24.203209 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:24.203237 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.204316 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.204322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:24.205701 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:24.205716 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.206496 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.206505 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:24.206749 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.207387 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:24.207412 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:24.210117 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:24.213321 (+ 3204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:24.213329 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:24.213334 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:24.213935 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:24.214353 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:24.214361 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:24.214367 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:24.214566 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:24.215094 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:24.215100 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:24.215321 (+ 221us) client_negotiation.cc:769] Sending connection context
0925 05:06:24.215518 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 05:06:24.215879 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260925 05:06:24.217550 22095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:06:24.218652 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.218218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:37501 (local address 127.0.0.1:41272)
0925 05:06:24.218399 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.218493 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 05:06:24.219628 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.219267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:37501 (local address 127.0.0.1:41286)
0925 05:06:24.219428 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.219511 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 05:06:24.219835 22095 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.60:37501: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
I20260925 05:06:24.261348 22169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } attempt: 3
I20260925 05:06:24.262722 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.262032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:37501 (local address 127.0.0.1:41296)
0925 05:06:24.262332 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.262484 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 05:06:24.264117 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.263523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:37501 (local address 127.0.0.1:41306)
0925 05:06:24.263817 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.263936 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
W20260925 05:06:24.264339 22169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.60:37501: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
I20260925 05:06:24.271065 22095 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } attempt: 1
I20260925 05:06:24.272120 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.271641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:37501 (local address 127.0.0.1:41316)
0925 05:06:24.271845 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.271946 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 05:06:24.273319 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.272849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:37501 (local address 127.0.0.1:41324)
0925 05:06:24.273037 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.273138 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 05:06:24.273561 22095 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.60:37501: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
I20260925 05:06:24.341807 22095 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } attempt: 2
I20260925 05:06:24.343143 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.342504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:37501 (local address 127.0.0.1:41326)
0925 05:06:24.342764 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.342910 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 05:06:24.344367 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.343945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:37501 (local address 127.0.0.1:41340)
0925 05:06:24.344122 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.344215 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 05:06:24.344630 22095 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.60:37501: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
W20260925 05:06:24.385399 22176 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:06:24.385761 22176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:24.385885 22176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:24.386018 22176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:24.393160 22176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:24.393282 22176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:24.393378 22176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:24.393473 22176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:24.393563 22176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:24.403144 22176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.247.20:38441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.247.62:45713,127.30.247.61:33401,127.30.247.60:37501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/krb5kdc/kudu_127.30.247.60.keytab
--principal=kudu/127.30.247.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.247.60:37501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.247.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:06:24.404385 22176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:24.405717 22176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:06:24.407574 22169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } attempt: 4
I20260925 05:06:24.408809 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.408272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:37501 (local address 127.0.0.1:41352)
0925 05:06:24.408491 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.408652 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 05:06:24.409762 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.409441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:37501 (local address 127.0.0.1:41364)
0925 05:06:24.409590 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.409667 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 05:06:24.409960 22169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.60:37501: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
W20260925 05:06:24.417973 22181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:24.418038 22184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:24.418074 22182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:24.418941 22176 server_base.cc:1034] running on GCE node
I20260925 05:06:24.419606 22176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:24.420889 22176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:24.422096 22176 hybrid_clock.cc:648] HybridClock initialized: now 1790312784422026 us; error 41 us; skew 500 ppm
I20260925 05:06:24.422474 22176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:06:24 dist-test-slave-5n86 krb5kdc[22015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312784, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:24.425552 22176 init.cc:377] Logged in from keytab as kudu/127.30.247.60@KRBTEST.COM (short username kudu)
I20260925 05:06:24.426986 22176 webserver.cc:460] Webserver started at http://127.30.247.60:45463/ using document root <none> and password file <none>
I20260925 05:06:24.427668 22176 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:24.427802 22176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:24.428128 22176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:24.431293 22176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-2/data/instance:
uuid: "c59d6acd9b8345b6b83b56ee8d4f5c1f"
format_stamp: "Formatted at 2026-09-25 05:06:24 on dist-test-slave-5n86"
server_key: "b93ed5d6caadb337297661c6a1685976"
server_key_iv: "830bcbc29e36389fc9fd447144f4729e"
server_key_version: "encryptionkey@0"
I20260925 05:06:24.432094 22176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-2/wal/instance:
uuid: "c59d6acd9b8345b6b83b56ee8d4f5c1f"
format_stamp: "Formatted at 2026-09-25 05:06:24 on dist-test-slave-5n86"
server_key: "b93ed5d6caadb337297661c6a1685976"
server_key_iv: "830bcbc29e36389fc9fd447144f4729e"
server_key_version: "encryptionkey@0"
I20260925 05:06:24.437022 22176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 05:06:24.440322 22191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:24.441527 22176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:06:24.441725 22176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-2/data,/tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-2/wal
uuid: "c59d6acd9b8345b6b83b56ee8d4f5c1f"
format_stamp: "Formatted at 2026-09-25 05:06:24 on dist-test-slave-5n86"
server_key: "b93ed5d6caadb337297661c6a1685976"
server_key_iv: "830bcbc29e36389fc9fd447144f4729e"
server_key_version: "encryptionkey@0"
I20260925 05:06:24.441965 22176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:24.450855 22095 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } attempt: 3
I20260925 05:06:24.451890 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.451396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:37501 (local address 127.0.0.1:41380)
0925 05:06:24.451618 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.451713 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 05:06:24.452970 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.452521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:37501 (local address 127.0.0.1:41386)
0925 05:06:24.452766 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.452849 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260925 05:06:24.453186 22095 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.247.60:37501: Network error: Client connection negotiation failed: client connection to 127.30.247.60:37501: connect: Connection refused (error 111)
I20260925 05:06:24.459975 22176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:24.463768 22176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:24.463976 22176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:24.481988 22176 rpc_server.cc:307] RPC server started. Bound to: 127.30.247.60:37501
I20260925 05:06:24.481994 22242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.247.60:37501 every 8 connection(s)
I20260925 05:06:24.483299 22176 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-2/data/info.pb
I20260925 05:06:24.486819 22243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:24.489614 31708 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43xS0j/build/asan/bin/kudu as pid 22176
I20260925 05:06:24.489805 31708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43xS0j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312430481417-31708-0/minicluster-data/master-2/wal/instance
I20260925 05:06:24.490257 31708 external_mini_cluster.cc:1428] Setting key 9314fffce087991d035c4bec8b42735c
I20260925 05:06:24.494534 22243 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45713 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:06:24 dist-test-slave-5n86 krb5kdc[22015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.60@KRBTEST.COM
Sep 25 05:06:24 dist-test-slave-5n86 krb5kdc[22015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312784, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.60@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:06:24.513409 22098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.497621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38148 (local address 127.30.247.62:45713)
0925 05:06:24.497875 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:24.497883 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:24.498570 (+ 687us) server_negotiation.cc:407] Connection header received
0925 05:06:24.498850 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:24.498861 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:24.498924 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:24.499033 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:24.500203 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.501303 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.503047 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.503565 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.503719 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.507052 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:24.507058 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:24.507065 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:24.507104 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:24.510293 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.511082 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.511086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.511089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.511258 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.511699 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.511704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.511708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.512061 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:24.512219 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:24.512802 (+ 583us) server_negotiation.cc:299] Negotiation successful
0925 05:06:24.513025 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 05:06:24.514344 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.492085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41392 (local address 127.30.247.60:37501)
0925 05:06:24.492894 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:24.492908 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:24.492956 (+ 48us) server_negotiation.cc:407] Connection header received
0925 05:06:24.495061 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:24.495090 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:24.495620 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:24.496097 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:24.497761 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.499853 (+ 2092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.501291 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.502386 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.502558 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.506704 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:24.506715 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:24.506726 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:24.506760 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:24.510229 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.511074 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.511081 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.511085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.511272 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.511699 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.511705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.511708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.512311 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:24.512627 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:24.512984 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 05:06:24.513342 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":177,"server-negotiator.queue_time_us":577,"thread_start_us":385,"threads_started":1}
I20260925 05:06:24.514391 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.497563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.62:45713 (local address 127.0.0.1:38148)
0925 05:06:24.498446 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.498490 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:24.498693 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:24.499091 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:24.499098 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:24.499716 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:24.500045 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:24.500058 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.501467 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.501472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:24.502914 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:24.502924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.503812 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.503819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:24.503982 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.504642 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:24.504662 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:24.506861 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:24.510416 (+ 3555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:24.510429 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:24.510441 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:24.510973 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:24.511382 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:24.511387 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:24.511391 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:24.511589 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:24.512182 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:24.512187 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:24.512353 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 05:06:24.512715 (+ 362us) client_negotiation.cc:240] Negotiation successful
0925 05:06:24.513207 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":2,"thread_start_us":513,"threads_started":1}
I20260925 05:06:24.516147 22243 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.61" port: 33401 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:06:24 dist-test-slave-5n86 krb5kdc[22015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312784, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.60@KRBTEST.COM for kudu/127.30.247.61@KRBTEST.COM
Sep 25 05:06:24 dist-test-slave-5n86 krb5kdc[22015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.62@KRBTEST.COM
I20260925 05:06:24.529166 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.516752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.61:33401 (local address 127.0.0.1:52952)
0925 05:06:24.516905 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.516926 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:24.517023 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:24.517317 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:24.517321 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:24.517700 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:24.517867 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:24.517879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.518884 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.518892 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:24.519755 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:24.519765 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.520485 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.520491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:24.520696 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.521235 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:24.521251 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:24.523209 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:24.526677 (+ 3468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:24.526685 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:24.526690 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:24.527162 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:24.527548 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:24.527554 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:24.527558 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:24.527698 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:24.528214 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:24.528219 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:24.528357 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 05:06:24.528519 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 05:06:24.528813 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 05:06:24.529222 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.516878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52952 (local address 127.30.247.61:33401)
0925 05:06:24.517034 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:24.517039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:24.517054 (+ 15us) server_negotiation.cc:407] Connection header received
0925 05:06:24.517106 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:24.517109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:24.517166 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:24.517242 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:24.518084 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.518746 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.519911 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.520338 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.520465 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.523407 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:24.523415 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:24.523420 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:24.523463 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:24.526543 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.527290 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.527297 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.527300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.527439 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.527807 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.527813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.527817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.528075 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:24.528242 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:24.528647 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 05:06:24.528851 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 05:06:24.530133 22243 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:06:24.532442 22098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.517899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38162 (local address 127.30.247.62:45713)
0925 05:06:24.518043 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:24.518050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:24.518067 (+ 17us) server_negotiation.cc:407] Connection header received
0925 05:06:24.518245 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:24.518250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:24.518317 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:24.518413 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:24.519508 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.520227 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.521682 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.522069 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.522230 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.526541 (+ 4311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:24.526545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:24.526548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:24.526575 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:24.529652 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.530394 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.530399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.530401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.530554 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.531048 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.531054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.531057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.531325 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:24.531617 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:24.531919 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 05:06:24.532141 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 05:06:24 dist-test-slave-5n86 krb5kdc[22015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312784, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.60@KRBTEST.COM for kudu/127.30.247.60@KRBTEST.COM
Sep 25 05:06:24 dist-test-slave-5n86 krb5kdc[22015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.247.61@KRBTEST.COM
I20260925 05:06:24.545315 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.530934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:37501 (local address 127.0.0.1:41400)
0925 05:06:24.531160 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.531175 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:24.531281 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:24.531548 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:24.531552 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:24.531910 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:24.532024 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:24.532032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.533171 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.533179 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:24.534409 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:24.534423 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.535401 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.535405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:24.535567 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.536112 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:24.536135 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:24.538676 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:24.541673 (+ 2997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:24.541679 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:24.541684 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:24.542227 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:24.542633 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:24.542641 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:24.542647 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:24.542795 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:24.544176 (+ 1381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:24.544179 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:24.544300 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 05:06:24.544467 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 05:06:24.544806 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260925 05:06:24.545405 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.530971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41400 (local address 127.30.247.60:37501)
0925 05:06:24.531123 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:24.531128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:24.531245 (+ 117us) server_negotiation.cc:407] Connection header received
0925 05:06:24.531379 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:24.531384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:24.531439 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:24.531534 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:24.532155 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.533051 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.534569 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.535252 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.535404 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.538852 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:24.538857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:24.538860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:24.538887 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:24.541531 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.542381 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.542386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.542394 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.542514 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.542905 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.542931 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.542938 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.543261 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:24.543411 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:24.544571 (+ 1160us) server_negotiation.cc:299] Negotiation successful
0925 05:06:24.545016 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":192,"server-negotiator.queue_time_us":56}
I20260925 05:06:24.550092 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.534091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52966 (local address 127.30.247.61:33401)
0925 05:06:24.534255 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:24.534262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:24.534280 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:06:24.534327 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:24.534333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:24.534400 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:24.534504 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:24.536062 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.537211 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.538989 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.539669 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.539802 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.544189 (+ 4387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:24.544194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:24.544197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:24.544225 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:24.547561 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.548142 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.548148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.548151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.548302 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.548711 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.548716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.548720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.549054 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:24.549314 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:24.549559 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 05:06:24.549761 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 05:06:24.551551 22243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c59d6acd9b8345b6b83b56ee8d4f5c1f: Bootstrap starting.
I20260925 05:06:24.554704 31708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:06:24.555063 22243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c59d6acd9b8345b6b83b56ee8d4f5c1f: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:24.556059 22243 log.cc:824] T 00000000000000000000000000000000 P c59d6acd9b8345b6b83b56ee8d4f5c1f: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:24.559159 22243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c59d6acd9b8345b6b83b56ee8d4f5c1f: No bootstrap required, opened a new log
I20260925 05:06:24.557065 22070 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e5857057b34d4b1da6685e0aa5ef97b0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38162
I20260925 05:06:24.569661 22243 raft_consensus.cc:348] T 00000000000000000000000000000000 P c59d6acd9b8345b6b83b56ee8d4f5c1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45713 } } peers { permanent_uuid: "35ed88e7a9524194b28f374016e47405" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 33401 } } peers { permanent_uuid: "c59d6acd9b8345b6b83b56ee8d4f5c1f" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } }
I20260925 05:06:24.570046 22243 raft_consensus.cc:729] T 00000000000000000000000000000000 P c59d6acd9b8345b6b83b56ee8d4f5c1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59d6acd9b8345b6b83b56ee8d4f5c1f, State: Initialized, Role: FOLLOWER
I20260925 05:06:24.570538 22243 consensus_queue.cc:260] T 00000000000000000000000000000000 P c59d6acd9b8345b6b83b56ee8d4f5c1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45713 } } peers { permanent_uuid: "35ed88e7a9524194b28f374016e47405" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 33401 } } peers { permanent_uuid: "c59d6acd9b8345b6b83b56ee8d4f5c1f" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } }
I20260925 05:06:24.571395 22249 sys_catalog.cc:455] T 00000000000000000000000000000000 P c59d6acd9b8345b6b83b56ee8d4f5c1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45713 } } peers { permanent_uuid: "35ed88e7a9524194b28f374016e47405" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 33401 } } peers { permanent_uuid: "c59d6acd9b8345b6b83b56ee8d4f5c1f" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } } }
I20260925 05:06:24.571734 22249 sys_catalog.cc:458] T 00000000000000000000000000000000 P c59d6acd9b8345b6b83b56ee8d4f5c1f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:06:24.572655 22243 sys_catalog.cc:564] T 00000000000000000000000000000000 P c59d6acd9b8345b6b83b56ee8d4f5c1f [sys.catalog]: configured and running, proceeding with master startup.
W20260925 05:06:24.582124 22260 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c59d6acd9b8345b6b83b56ee8d4f5c1f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:06:24.582278 22260 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:06:24.580019 22176 master_runner.cc:186] Error getting master registration for 127.30.247.62:45713: OK, instance_id { permanent_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" instance_seqno: 1790312783488390 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 05:06:24.591063 22176 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 05:06:24.630393 22095 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } attempt: 4
Sep 25 05:06:24 dist-test-slave-5n86 krb5kdc[22015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312783, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.62@KRBTEST.COM for kudu/127.30.247.60@KRBTEST.COM
I20260925 05:06:24.647439 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.631190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:37501 (local address 127.0.0.1:41416)
0925 05:06:24.631409 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.631432 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:24.631564 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:24.631961 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:24.631967 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:24.632475 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:24.632745 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:24.632757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.634195 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.634201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:24.635702 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:24.635718 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.636733 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.636740 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:24.636947 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.637629 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:24.637651 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:24.640446 (+ 2795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:24.644424 (+ 3978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:24.644430 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:24.644434 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:24.644974 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:24.645377 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:24.645382 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:24.645386 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:24.645573 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:24.646364 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:24.646369 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:24.646578 (+ 209us) client_negotiation.cc:769] Sending connection context
0925 05:06:24.646756 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 05:06:24.647027 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 05:06:24.647468 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.631284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41416 (local address 127.30.247.60:37501)
0925 05:06:24.631491 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:24.631498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:24.631530 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:06:24.631707 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:24.631713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:24.631797 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:24.631912 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:24.632897 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.634048 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.635855 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.636566 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.636694 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.640650 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:24.640663 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:24.640669 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:24.640706 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:24.644298 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.645101 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.645108 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.645115 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.645255 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.645687 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.645693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.645696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.646147 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:24.646297 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:24.646877 (+ 580us) server_negotiation.cc:299] Negotiation successful
0925 05:06:24.647108 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 05:06:24.652010 22095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0: Bootstrap starting.
I20260925 05:06:24.655956 22095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:24.657088 22095 log.cc:824] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:24.659762 22095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0: No bootstrap required, opened a new log
I20260925 05:06:24.667366 22095 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45713 } } peers { permanent_uuid: "35ed88e7a9524194b28f374016e47405" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 33401 } } peers { permanent_uuid: "c59d6acd9b8345b6b83b56ee8d4f5c1f" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } }
I20260925 05:06:24.667779 22095 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5857057b34d4b1da6685e0aa5ef97b0, State: Initialized, Role: FOLLOWER
I20260925 05:06:24.668292 22095 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45713 } } peers { permanent_uuid: "35ed88e7a9524194b28f374016e47405" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 33401 } } peers { permanent_uuid: "c59d6acd9b8345b6b83b56ee8d4f5c1f" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } }
I20260925 05:06:24.669312 22264 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45713 } } peers { permanent_uuid: "35ed88e7a9524194b28f374016e47405" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 33401 } } peers { permanent_uuid: "c59d6acd9b8345b6b83b56ee8d4f5c1f" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } } }
I20260925 05:06:24.669701 22264 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:06:24.670444 22095 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:06:24.670557 22070 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 05:06:24.670758 22070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:24.674281 22070 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45713 } } peers { permanent_uuid: "35ed88e7a9524194b28f374016e47405" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 33401 } } peers { permanent_uuid: "c59d6acd9b8345b6b83b56ee8d4f5c1f" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } }
I20260925 05:06:24.676342 22070 leader_election.cc:288] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0 [CANDIDATE]: Term 1 election: Requested vote from peers 35ed88e7a9524194b28f374016e47405 (127.30.247.61:33401), c59d6acd9b8345b6b83b56ee8d4f5c1f (127.30.247.60:37501)
I20260925 05:06:24.676942 22218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c59d6acd9b8345b6b83b56ee8d4f5c1f"
I20260925 05:06:24.677457 22218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c59d6acd9b8345b6b83b56ee8d4f5c1f [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:24.681111 22218 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c59d6acd9b8345b6b83b56ee8d4f5c1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5857057b34d4b1da6685e0aa5ef97b0 in term 1.
I20260925 05:06:24.682405 22044 leader_election.cc:302] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c59d6acd9b8345b6b83b56ee8d4f5c1f, e5857057b34d4b1da6685e0aa5ef97b0; no voters:
I20260925 05:06:24.682956 22264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:24.689291 22264 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0 [term 1 LEADER]: Becoming Leader. State: Replica: e5857057b34d4b1da6685e0aa5ef97b0, State: Running, Role: LEADER
I20260925 05:06:24.690511 22264 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45713 } } peers { permanent_uuid: "35ed88e7a9524194b28f374016e47405" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 33401 } } peers { permanent_uuid: "c59d6acd9b8345b6b83b56ee8d4f5c1f" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } }
I20260925 05:06:24.704828 22275 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5857057b34d4b1da6685e0aa5ef97b0. Latest consensus state: current_term: 1 leader_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45713 } } peers { permanent_uuid: "35ed88e7a9524194b28f374016e47405" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 33401 } } peers { permanent_uuid: "c59d6acd9b8345b6b83b56ee8d4f5c1f" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } } }
I20260925 05:06:24.705246 22275 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:24.709137 22169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } attempt: 5
W20260925 05:06:24.716023 22197 tablet.cc:2369] T 00000000000000000000000000000000 P c59d6acd9b8345b6b83b56ee8d4f5c1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:06:24.718575 22098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.680725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38172 (local address 127.30.247.62:45713)
0925 05:06:24.680906 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:24.680912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:24.689290 (+ 8378us) server_negotiation.cc:407] Connection header received
0925 05:06:24.689618 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:24.689628 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:24.689737 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:24.689825 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:24.692706 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.693793 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.700645 (+ 6852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.701532 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.701657 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.708692 (+ 7035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:24.708699 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:24.708706 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:24.708749 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:24.713569 (+ 4820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.714582 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.714591 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.714598 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.714809 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.715366 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.715374 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.715381 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.717057 (+ 1676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:24.717372 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:24.717925 (+ 553us) server_negotiation.cc:299] Negotiation successful
0925 05:06:24.718180 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:06:24.719704 22282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:06:24.722381 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.680909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41418 (local address 127.30.247.60:37501)
0925 05:06:24.681137 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:24.681143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:24.692997 (+ 11854us) server_negotiation.cc:407] Connection header received
0925 05:06:24.693919 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:24.693928 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:24.694045 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:24.694125 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:24.698114 (+ 3989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.699200 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.703617 (+ 4417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.704329 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.704438 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.710700 (+ 6262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:24.710709 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:24.710717 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:24.710769 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:24.716390 (+ 5621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.717797 (+ 1407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.717805 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.717809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.718038 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.719116 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.719122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.719125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.719839 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:24.720232 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:24.721179 (+ 947us) server_negotiation.cc:299] Negotiation successful
0925 05:06:24.721598 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260925 05:06:24.728142 22049 tablet.cc:2369] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 05:06:24 dist-test-slave-5n86 krb5kdc[22015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312783, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.247.61@KRBTEST.COM for kudu/127.30.247.60@KRBTEST.COM
I20260925 05:06:24.733831 22282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:06:24.735309 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.693572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52972 (local address 127.30.247.61:33401)
0925 05:06:24.693785 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:24.693793 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:24.702589 (+ 8796us) server_negotiation.cc:407] Connection header received
0925 05:06:24.703006 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:24.703013 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:24.703106 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:24.703184 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:24.706723 (+ 3539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.707813 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.717548 (+ 9735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.718187 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.718288 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.725282 (+ 6994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:24.725290 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:24.725295 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:24.725332 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:24.729009 (+ 3677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.732131 (+ 3122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.732137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.732141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.732280 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.733133 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.733138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.733142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.733505 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:24.733732 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:24.734689 (+ 957us) server_negotiation.cc:299] Negotiation successful
0925 05:06:24.734942 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 05:06:24.741700 22287 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 05:06:24.744441 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.709704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.247.60:37501 (local address 127.0.0.1:41424)
0925 05:06:24.710054 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.710072 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:24.710173 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:24.723486 (+ 13313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:24.723493 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:24.723983 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:24.724184 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:24.724199 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.726550 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.726556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:24.728092 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:24.728111 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.729669 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.729675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:24.729927 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.730507 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:24.730544 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:24.733484 (+ 2940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:24.740852 (+ 7368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:24.740861 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:24.740867 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:24.741489 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:24.742252 (+ 763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:24.742256 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:24.742262 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:24.742417 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:24.743498 (+ 1081us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:24.743503 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:24.743671 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 05:06:24.743809 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 05:06:24.744071 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231}
I20260925 05:06:24.746148 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.710370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41424 (local address 127.30.247.60:37501)
0925 05:06:24.722988 (+ 12618us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:24.722993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:24.723035 (+ 42us) server_negotiation.cc:407] Connection header received
0925 05:06:24.723116 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:24.723121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:24.723252 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:24.724374 (+ 1122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:24.724586 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.726324 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.728362 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.729422 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.730286 (+ 864us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.733833 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:24.733841 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:24.733848 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:24.733914 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:24.740598 (+ 6684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.741775 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.741782 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.741787 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.742087 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:24.742644 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:24.742678 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:24.742681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:24.743306 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:24.744749 (+ 1443us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:24.745004 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 05:06:24.745404 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12436,"thread_start_us":28862,"threads_started":1}
I20260925 05:06:24.760030 22218 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c59d6acd9b8345b6b83b56ee8d4f5c1f [term 1 FOLLOWER]: Refusing update from remote peer e5857057b34d4b1da6685e0aa5ef97b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:24.760990 22275 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59d6acd9b8345b6b83b56ee8d4f5c1f" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:24.762471 22169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35ed88e7a9524194b28f374016e47405: Bootstrap starting.
I20260925 05:06:24.768915 22169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35ed88e7a9524194b28f374016e47405: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:24.747571 22028 master_runner.cc:186] Error getting master registration for 127.30.247.61:33401: OK, instance_id { permanent_uuid: "35ed88e7a9524194b28f374016e47405" instance_seqno: 1790312783980465 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 05:06:24.772364 22169 log.cc:824] T 00000000000000000000000000000000 P 35ed88e7a9524194b28f374016e47405: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:24.773147 22028 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 05:06:24.779350 22169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35ed88e7a9524194b28f374016e47405: No bootstrap required, opened a new log
I20260925 05:06:24.784456 22249 sys_catalog.cc:455] T 00000000000000000000000000000000 P c59d6acd9b8345b6b83b56ee8d4f5c1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5857057b34d4b1da6685e0aa5ef97b0. Latest consensus state: current_term: 1 leader_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45713 } } peers { permanent_uuid: "35ed88e7a9524194b28f374016e47405" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 33401 } } peers { permanent_uuid: "c59d6acd9b8345b6b83b56ee8d4f5c1f" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } } }
I20260925 05:06:24.784823 22249 sys_catalog.cc:458] T 00000000000000000000000000000000 P c59d6acd9b8345b6b83b56ee8d4f5c1f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:06:24.790787 22169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35ed88e7a9524194b28f374016e47405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45713 } } peers { permanent_uuid: "35ed88e7a9524194b28f374016e47405" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 33401 } } peers { permanent_uuid: "c59d6acd9b8345b6b83b56ee8d4f5c1f" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } }
I20260925 05:06:24.791229 22169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35ed88e7a9524194b28f374016e47405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ed88e7a9524194b28f374016e47405, State: Initialized, Role: FOLLOWER
I20260925 05:06:24.791373 22288 mvcc.cc:205] Tried to move back new op lower bound from 7333121166365040640 to 7333121166106488832. Current Snapshot: MvccSnapshot[applied={T|T < 7333121166365040640}]
I20260925 05:06:24.791822 22169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35ed88e7a9524194b28f374016e47405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45713 } } peers { permanent_uuid: "35ed88e7a9524194b28f374016e47405" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 33401 } } peers { permanent_uuid: "c59d6acd9b8345b6b83b56ee8d4f5c1f" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } }
I20260925 05:06:24.792714 22275 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45713 } } peers { permanent_uuid: "35ed88e7a9524194b28f374016e47405" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 33401 } } peers { permanent_uuid: "c59d6acd9b8345b6b83b56ee8d4f5c1f" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } } }
I20260925 05:06:24.792992 22275 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:24.796896 22282 catalog_manager.cc:1347] Generated new cluster ID: ff80afa6966444af83761e19dfa3e7cb
I20260925 05:06:24.797001 22282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:06:24.800493 22249 sys_catalog.cc:455] T 00000000000000000000000000000000 P c59d6acd9b8345b6b83b56ee8d4f5c1f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45713 } } peers { permanent_uuid: "35ed88e7a9524194b28f374016e47405" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 33401 } } peers { permanent_uuid: "c59d6acd9b8345b6b83b56ee8d4f5c1f" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } } }
I20260925 05:06:24.801302 22293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35ed88e7a9524194b28f374016e47405 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45713 } } peers { permanent_uuid: "35ed88e7a9524194b28f374016e47405" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 33401 } } peers { permanent_uuid: "c59d6acd9b8345b6b83b56ee8d4f5c1f" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } } }
I20260925 05:06:24.801702 22293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35ed88e7a9524194b28f374016e47405 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:06:24.801919 22249 sys_catalog.cc:458] T 00000000000000000000000000000000 P c59d6acd9b8345b6b83b56ee8d4f5c1f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:06:24.802644 22169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35ed88e7a9524194b28f374016e47405 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:06:24.803068 22143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35ed88e7a9524194b28f374016e47405 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:24.806572 22143 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 35ed88e7a9524194b28f374016e47405 [term 1 FOLLOWER]: Refusing update from remote peer e5857057b34d4b1da6685e0aa5ef97b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:24.807178 22144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "35ed88e7a9524194b28f374016e47405"
I20260925 05:06:24.807386 22264 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35ed88e7a9524194b28f374016e47405" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:24.810511 22144 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 35ed88e7a9524194b28f374016e47405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5857057b34d4b1da6685e0aa5ef97b0 in term 1.
I20260925 05:06:24.823979 22282 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:06:24.825810 22282 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:06:24.839805 22282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0: Generated new TSK 0
I20260925 05:06:24.840837 22282 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:06:24.849129 22293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35ed88e7a9524194b28f374016e47405 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5857057b34d4b1da6685e0aa5ef97b0. Latest consensus state: current_term: 1 leader_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45713 } } peers { permanent_uuid: "35ed88e7a9524194b28f374016e47405" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 33401 } } peers { permanent_uuid: "c59d6acd9b8345b6b83b56ee8d4f5c1f" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } } }
I20260925 05:06:24.849567 22293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35ed88e7a9524194b28f374016e47405 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:06:24.849094 22300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35ed88e7a9524194b28f374016e47405 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45713 } } peers { permanent_uuid: "35ed88e7a9524194b28f374016e47405" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 33401 } } peers { permanent_uuid: "c59d6acd9b8345b6b83b56ee8d4f5c1f" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } } }
I20260925 05:06:24.849829 22300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35ed88e7a9524194b28f374016e47405 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:06:24.851048 22264 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5857057b34d4b1da6685e0aa5ef97b0" member_type: VOTER last_known_addr { host: "127.30.247.62" port: 45713 } } peers { permanent_uuid: "35ed88e7a9524194b28f374016e47405" member_type: VOTER last_known_addr { host: "127.30.247.61" port: 33401 } } peers { permanent_uuid: "c59d6acd9b8345b6b83b56ee8d4f5c1f" member_type: VOTER last_known_addr { host: "127.30.247.60" port: 37501 } } }
I20260925 05:06:24.851351 22264 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5857057b34d4b1da6685e0aa5ef97b0 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:24.910111 22311 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260925 05:06:24.939744 22098 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:24.930337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38188 (local address 127.30.247.62:45713)
0925 05:06:24.930634 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:24.930643 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:24.937132 (+ 6489us) server_negotiation.cc:407] Connection header received
0925 05:06:24.938109 (+ 977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:24.938116 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:24.938220 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:24.938304 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:24.939483 (+ 1179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38188: BlockingRecv error: recv got EOF from 127.0.0.1:38188 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260925 05:06:24.941555 22246 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:24.930410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41432 (local address 127.30.247.60:37501)
0925 05:06:24.930588 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:24.930594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:24.937679 (+ 7085us) server_negotiation.cc:407] Connection header received
0925 05:06:24.940075 (+ 2396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:24.940081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:24.940181 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:24.940261 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:24.941342 (+ 1081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41432: BlockingRecv error: recv got EOF from 127.0.0.1:41432 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 05:06:24.943902 22173 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:24.937307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52984 (local address 127.30.247.61:33401)
0925 05:06:24.937495 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:24.937501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:24.942369 (+ 4868us) server_negotiation.cc:407] Connection header received
0925 05:06:24.942641 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:24.942647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:24.942735 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:24.942803 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:24.943695 (+ 892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52984: BlockingRecv error: recv got EOF from 127.0.0.1:52984 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 05:06:24.950598 22173 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:24.948836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52986 (local address 127.30.247.61:33401)
0925 05:06:24.948998 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:24.949006 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:24.949117 (+ 111us) server_negotiation.cc:407] Connection header received
0925 05:06:24.949408 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:24.949414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:24.949535 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:24.949647 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:24.950352 (+ 705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52986: BlockingRecv error: recv got EOF from 127.0.0.1:52986 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 05:06:24.956826 22173 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:24.954927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52994 (local address 127.30.247.61:33401)
0925 05:06:24.955073 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:24.955078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:24.955513 (+ 435us) server_negotiation.cc:407] Connection header received
0925 05:06:24.955688 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:24.955695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:24.955777 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:24.955878 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:24.956636 (+ 758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52994: BlockingRecv error: recv got EOF from 127.0.0.1:52994 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 05:06:24.964160 22173 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:24.962225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53006 (local address 127.30.247.61:33401)
0925 05:06:24.962372 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:24.962377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:24.962850 (+ 473us) server_negotiation.cc:407] Connection header received
0925 05:06:24.963018 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:24.963024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:24.963116 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:24.963226 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:24.963928 (+ 702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53006: BlockingRecv error: recv got EOF from 127.0.0.1:53006 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 05:06:24.973423 22173 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:24.971262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53020 (local address 127.30.247.61:33401)
0925 05:06:24.971457 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:24.971462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:24.971937 (+ 475us) server_negotiation.cc:407] Connection header received
0925 05:06:24.972130 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:24.972136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:24.972224 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:24.972325 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:24.973208 (+ 883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53020: BlockingRecv error: recv got EOF from 127.0.0.1:53020 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260925 05:06:25.004321 22173 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:24.985632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53026 (local address 127.30.247.61:33401)
0925 05:06:24.985815 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:24.985822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:25.002455 (+ 16633us) server_negotiation.cc:407] Connection header received
0925 05:06:25.002794 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:25.002801 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:25.002905 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:25.002981 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:25.004057 (+ 1076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53026: BlockingRecv error: recv got EOF from 127.0.0.1:53026 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 05:06:25.029529 22173 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:25.027167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53034 (local address 127.30.247.61:33401)
0925 05:06:25.027360 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:25.027368 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:25.027954 (+ 586us) server_negotiation.cc:407] Connection header received
0925 05:06:25.028163 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:25.028169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:25.028283 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:25.028398 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:25.029294 (+ 896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53034: BlockingRecv error: recv got EOF from 127.0.0.1:53034 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 05:06:25.074260 22173 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:25.071779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53048 (local address 127.30.247.61:33401)
0925 05:06:25.071986 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:25.072010 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:25.072613 (+ 603us) server_negotiation.cc:407] Connection header received
0925 05:06:25.072891 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:25.072898 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:25.073003 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:25.073130 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:25.073993 (+ 863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53048: BlockingRecv error: recv got EOF from 127.0.0.1:53048 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260925 05:06:25.144732 22173 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:25.142808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53056 (local address 127.30.247.61:33401)
0925 05:06:25.143095 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:25.143102 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:25.143363 (+ 261us) server_negotiation.cc:407] Connection header received
0925 05:06:25.143527 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:25.143533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:25.143637 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:25.143766 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:25.144490 (+ 724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53056: BlockingRecv error: recv got EOF from 127.0.0.1:53056 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
W20260925 05:06:25.278506 22173 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:25.276616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53068 (local address 127.30.247.61:33401)
0925 05:06:25.276887 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:25.276893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:25.277139 (+ 246us) server_negotiation.cc:407] Connection header received
0925 05:06:25.277298 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:25.277302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:25.277378 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:25.277466 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:25.278230 (+ 764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53068: BlockingRecv error: recv got EOF from 127.0.0.1:53068 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260925 05:06:25.539891 22173 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:25.538053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53074 (local address 127.30.247.61:33401)
0925 05:06:25.538231 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:25.538236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:25.538750 (+ 514us) server_negotiation.cc:407] Connection header received
0925 05:06:25.538918 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:25.538923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:25.539009 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:25.539108 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:25.539732 (+ 624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53074: BlockingRecv error: recv got EOF from 127.0.0.1:53074 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 05:06:25.584216 22260 catalog_manager.cc:1259] Loaded cluster ID: ff80afa6966444af83761e19dfa3e7cb
I20260925 05:06:25.584332 22260 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c59d6acd9b8345b6b83b56ee8d4f5c1f: loading cluster ID for follower catalog manager: success
I20260925 05:06:25.586687 22260 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c59d6acd9b8345b6b83b56ee8d4f5c1f: acquiring CA information for follower catalog manager: success
I20260925 05:06:25.588349 22260 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c59d6acd9b8345b6b83b56ee8d4f5c1f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:06:25.914486 22311 catalog_manager.cc:1259] Loaded cluster ID: ff80afa6966444af83761e19dfa3e7cb
I20260925 05:06:25.914678 22311 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 35ed88e7a9524194b28f374016e47405: loading cluster ID for follower catalog manager: success
I20260925 05:06:25.917421 22311 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 35ed88e7a9524194b28f374016e47405: acquiring CA information for follower catalog manager: success
I20260925 05:06:25.919166 22311 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 35ed88e7a9524194b28f374016e47405: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:06:26.063817 22376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:26.055550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53088 (local address 127.30.247.61:33401)
0925 05:06:26.057052 (+ 1502us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:26.057059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:26.057392 (+ 333us) server_negotiation.cc:407] Connection header received
0925 05:06:26.057569 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:26.057575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:26.057785 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:26.057954 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:26.058337 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:26.059549 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:26.061013 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:26.061721 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:26.061844 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:26.062354 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:26.063153 (+ 799us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:26.063295 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:26.063405 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 05:06:26.063496 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1337,"thread_start_us":1147,"threads_started":1}
I20260925 05:06:26.077098 22382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:26.069978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41444 (local address 127.30.247.60:37501)
0925 05:06:26.070606 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:26.070612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:26.070913 (+ 301us) server_negotiation.cc:407] Connection header received
0925 05:06:26.071124 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:26.071129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:26.071373 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:26.071569 (+ 196us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:26.072016 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:26.072938 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:26.074854 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:26.075255 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:26.075409 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:26.075942 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:26.076455 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:26.076622 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:26.076714 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 05:06:26.076779 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":345,"threads_started":1}
I20260925 05:06:26.107615 22382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:26.084054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41458 (local address 127.30.247.60:37501)
0925 05:06:26.084214 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:26.084219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:26.098801 (+ 14582us) server_negotiation.cc:407] Connection header received
0925 05:06:26.098992 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:26.098997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:26.099091 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:26.099197 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:26.099876 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:26.101337 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:26.104763 (+ 3426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:26.105608 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:26.105723 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:26.106584 (+ 861us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:26.106738 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:26.106810 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:26.107081 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 05:06:26.107200 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 05:06:26.109723 22388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:26.083789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38200 (local address 127.30.247.62:45713)
0925 05:06:26.096359 (+ 12570us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:26.096366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:26.097072 (+ 706us) server_negotiation.cc:407] Connection header received
0925 05:06:26.097341 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:26.097348 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:26.097560 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:26.097726 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:26.098404 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:26.099515 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:26.101680 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:26.102319 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:26.102436 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:26.103292 (+ 856us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:26.103727 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:26.103832 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:26.108458 (+ 4626us) server_negotiation.cc:299] Negotiation successful
0925 05:06:26.109424 (+ 966us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12425,"thread_start_us":12582,"threads_started":1}
I20260925 05:06:26.111330 22376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:26.099403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53100 (local address 127.30.247.61:33401)
0925 05:06:26.099591 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:26.099597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:26.104395 (+ 4798us) server_negotiation.cc:407] Connection header received
0925 05:06:26.104699 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:26.104704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:26.104786 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:26.104844 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:26.105376 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:26.106497 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:26.108632 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:26.109261 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:26.109352 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:26.110394 (+ 1042us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:26.110504 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:26.110576 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:26.110968 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 05:06:26.111058 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 05:06:26.115355 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 22028
I20260925 05:06:26.117098 22090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:06:26.239979 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 22028
I20260925 05:06:26.268335 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 22102
I20260925 05:06:26.269735 22164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:06:26.387454 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 22102
I20260925 05:06:26.403537 31708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task43xS0j/build/asan/bin/kudu pid 22176
I20260925 05:06:26.404649 22238 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:06:26.508805 31708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43xS0j/build/asan/bin/kudu with pid 22176
2026-09-25T05:06:26Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4814 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9060 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (355996 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST